Finding a lost song
with Node.js & async iterators

Luciano Mammino (@loige)

Sailsconf 2021
2021-06-24

loige.link/iter-sails

Get these slides!

loige

Photo by Darius Bashar on Unsplash

A random song you haven't listened to
in years pops into your head...

It doesn't matter what you do all day...
It keeps coming back to you!

Photo by Attentie Attentie on Unsplash

And now you want to listen to it!

Photo by Volodymyr Hryshchenko on Unsplash

But, what if you can't remember

the title or the author?!

Photo by Tachina Lee on Unsplash

THERE MUST BE A WAY TO REMEMBER!

Photo by Marius Niveri on Unsplash

Today, I'll tell you how I solved this problem using

- Last.fm API

- Node.js

- Async Iterators

Photo by Quinton Coetzee on Unsplash

Let me introduce myself first...

I'm Luciano (🇮🇹🍕🍝) 👋

Senior Architect @ fourTheorem (Dublin 🇮🇪) 👨‍💻

nodejsdp.link

Co-Author of Node.js Design Patterns 👉

Connect with me:

loige.co (blog)

@loige (twitter)

loige (twitch)

lmammino (github)

We are business focused technologists that deliver.

Accelerated Serverless | AI as a Service | Platform Modernisation

We are hiring: do you want to work with us?

loige

There was this song in my mind...

loige

I could only remember some random parts and the word "dark" (probably in the title)

loige

Luciano - scrobbling since 12 Feb 2007

loige

~250k scrobbles... that song must be there!

loige

~5k pages of history &
no search functionality! 😓

loige

But there's an API!

https://www.last.fm/api

loige

Let's give it a shot

curl "http://ws.audioscrobbler.com/2.0/?method=user.getrecenttracks&user=loige&api_key=${API_KEY}&format=json" | jq .

loige

It works! 🥳

Let's convert this to JavaScript

loige

import querystring from 'querystring'
import axios from 'axios'

const query = querystring.stringify({
  method: 'user.getrecenttracks',
  user: 'loige',
  api_key: process.env.API_KEY,
  format: 'json'
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`

const response = await axios.get(url)

console.log(response.data)

loige

We are getting a "paginated" response with 50 tracks per page

but there are 51 here! 🤔

How do we fetch the next pages?

(let's ignore this for now...)

loige

let page = 1
while (true) {
  const query = querystring.stringify({
    method: 'user.getrecenttracks',
    user: 'loige',
    api_key: process.env.API_KEY,
    format: 'json',
    page
  })
  const url = `https://ws.audioscrobbler.com/2.0/?${query}`

  const response = await axios.get(url)

  console.log(response.data)

  if (page === Number(response.data.recenttracks['@attr'].totalPages)) {
    break // it's the last page!
  }

  page++
}

loige

Seems good!

Let's look at the tracks...

loige

// ...
for (const track of response.data.recenttracks.track) {
  console.log(
    track.date?.['#text'],
    `${track.artist['#text']} - ${track.name}`
  )
}
console.log('--- end page ---')
// ...

loige

* Note that page size here is 10 tracks per page

Every page has a song with undefined time...

This is the song I am currently listening to!

It appears at the top of every page.

Sometimes there are duplicated tracks between pages... 😨

The "sliding windows" problem 😩

loige

...

tracks (newest to oldest)

Page1

Page2

...

Page1

Page2

new track

moved from page 1 to page 2

loige

Time based windows 😎

loige

...*

tracks (newest to oldest)

Page1

before t1

(page 1 "to" t1)

before t2

(page 1 "to" t2)

* we are done when we get an empty page (or num pages is 1)

let to
while (true) {
  const query = querystring.stringify({
    method: 'user.getrecenttracks',
    user: 'loige',
    api_key: process.env.API_KEY,
    format: 'json',
    limit: '10',
    to
  })
  const url = `https://ws.audioscrobbler.com/2.0/?${query}`

  const response = await axios.get(url)

  const tracks = response.data.recenttracks.track

  console.log(
    `--- ↓ page to ${to}`,
    `remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---`
  )

  for (const track of tracks) {
    console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`)
  }

  if (response.data.recenttracks['@attr'].totalPages <= 1) {
    break // it's the last page!
  }

  const lastTrackInPage = tracks[tracks.length - 1]
  to = lastTrackInPage.date.uts
}

loige

The track of the last timestamp becomes the boundary for the next page

We have a working solution! 🎉
Can we generalise it?

loige

We know how to iterate over every page/track.
How do we expose this information?

loige

const reader = LastFmRecentTracks({
  apikey: process.env.API_KEY,
  user: 'loige'
})

// callbacks

reader.readPages(
  (page) => { /* ... */ }, // on page
  (err) => { /* ... */} // on completion (or error)
)

loige

const reader = LastFmRecentTracks({
  apikey: process.env.API_KEY,
  user: 'loige'
})

// event emitter

reader.read()
reader.on('page', (page) => { /* ... */ })
reader.on('completed', (err) => { /* ... */ })

loige

const reader = LastFmRecentTracks({
  apikey: process.env.API_KEY,
  user: 'loige'
})

// streams <3

reader.pipe(/* transform or writable stream here */)
reader.on('end', () => { /* ... */ })
reader.on('error', () => { /* ... */ })

loige

import { pipeline } from 'stream'

const reader = LastFmRecentTracks({
  apikey: process.env.API_KEY,
  user: 'loige'
})

// streams pipeline <3 <3
pipeline(
  reader,
  yourProcessingStream,
  (err) => {
    // handle completion or err
  }
)

loige

const reader = LastFmRecentTracks({
  apikey: process.env.API_KEY,
  user: 'loige'
})

// ASYNC ITERATORS!


for await (const page of reader) {
  /* ... */
}

// ... do more stuff when all the data is consumed

loige

const reader = LastFmRecentTracks({
  apikey: process.env.API_KEY,
  user: 'loige'
})

// ASYNC ITERATORS WITH ERROR HANDLING!

try {
  for await (const page of reader) {
    /* ... */
  }
} catch (err) {
  // handle errors
}
// ... do more stuff when all the data is consumed

loige

How can we build an async iterator? 🧐

loige

Meet the iteration protocols!

loige

loige.co/javascript-iterator-patterns

The iterator protocol

An object is an iterator if it has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value.

loige

function createCountdown (from) {
  let nextVal = from
  return {
    next () {
      if (nextVal < 0) {
        return { done: true }
      }

      return { 
        done: false,
        value: nextVal--
      }
    }
  }
}

loige

const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }

console.log(countdown.next())
// { done: false, value: 2 }

console.log(countdown.next())
// { done: false, value: 1 }

console.log(countdown.next())
// { done: false, value: 0 }

console.log(countdown.next())
// { done: true }

loige

Generator functions "produce" iterators!

loige

function * createCountdown (from) {
  for (let i = from; i >= 0; i--) {
    yield i
  }
}

loige

const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }

console.log(countdown.next())
// { done: false, value: 2 }

console.log(countdown.next())
// { done: false, value: 1 }

console.log(countdown.next())
// { done: false, value: 0 }

console.log(countdown.next())
// { done: true, value: undefined }

loige

The iterable protocol

An object is iterable if it implements the @@iterator* method, a zero-argument function that returns an iterator.

loige

*Symbol.iterator

function createCountdown (from) {
  let nextVal = from
  return {
    [Symbol.iterator]: () => ({
      next () {
        if (nextVal < 0) {
          return { done: true }
        }

        return { done: false, value: nextVal-- }
      }
    })
  }
}

loige

function createCountdown (from) {
  return {
    [Symbol.iterator]: function * () {
      for (let i = from; i >= 0; i--) {
        yield i
      }
    }
  }
}

loige

const countdown = createCountdown(3)

for (const value of countdown) {
  console.log(value)
}

// 3
// 2
// 1
// 0

loige

OK. So far this is all synchronous iteration.
What about async? 🙄

loige

The async iterator protocol

An object is an async iterator if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value.

loige

import { setTimeout } from 'timers/promises'

function createAsyncCountdown (from, delay = 1000) {
  let nextVal = from
  return {
    async next () {
      await setTimeout(delay)
      if (nextVal < 0) {
        return { done: true }
      }

      return { done: false, value: nextVal-- }
    }
  }
}

loige

const countdown = createAsyncCountdown(3)
console.log(await countdown.next())
// { done: false, value: 3 }

console.log(await countdown.next())
// { done: false, value: 2 }

console.log(await countdown.next())
// { done: false, value: 1 }

console.log(await countdown.next())
// { done: false, value: 0 }

console.log(await countdown.next())
// { done: true }

loige

import { setTimeout } from 'timers/promises'

// async generators "produce" async iterators!

async function * createAsyncCountdown (from, delay = 1000) {
  for (let i = from; i >= 0; i--) {
    await setTimeout(delay)
    yield i
  }
}

loige

The async iterable protocol

An object is an async iterable if it implements the @@asyncIterator* method, a zero-argument function that returns an async iterator.

loige

*Symbol.asyncIterator

import { setTimeout } from 'timers/promises'

function createAsyncCountdown (from, delay = 1000) {
  return {
    [Symbol.asyncIterator]: async function * () {
      for (let i = from; i >= 0; i--) {
        await setTimeout(delay)
        yield i
      }
    }
  }
}

loige

const countdown = createAsyncCountdown(3)

for await (const value of countdown) {
  console.log(value)
}

loige

Now we know how to make our LastFmRecentTracks an Async Iterable 🤩

loige

function createLastFmRecentTracks (apiKey, user) {
  return {
    [Symbol.asyncIterator]: async function * () {
      let to
      while (true) {
        const query = querystring.stringify({
          method: 'user.getrecenttracks',
          user,
          api_key: apiKey,
          format: 'json',
          to
        })
        const url = `https://ws.audioscrobbler.com/2.0/?${query}`

        const response = await axios.get(url)

        const tracks = response.data.recenttracks.track

        yield tracks

        if (response.data.recenttracks['@attr'].totalPages <= 1) {
          break // it's the last page!
        }

        const lastTrackInPage = tracks[tracks.length - 1]
        to = lastTrackInPage.date.uts
      }
    }
  }
}

loige

const recentTracks = createLastFmRecentTracks(
  process.env.API_KEY,
  'loige'
)

for await (const page of recentTracks) {
  console.log(page)
}

loige

Let's search for all the songs that contain the word "dark" in their title! 🧐

loige

async function main () {
  const recentTracks = createLastFmRecentTracks(
    process.env.API_KEY,
    'loige'
  )
  
  for await (const page of recentTracks) {
    for (const track of page) {
      if (track.name.toLowerCase().includes('dark')) {
        console.log(`${track.artist['#text']} - ${track.name}`)
      }
    }
  }
}

loige

OMG! This is the song! 😱
...from 8 years ago!

For a more serious package that allows you to fetch data from Last.fm:

loige

npm install scrobbles

Cover picture by Eric Nopanen on Unsplash
Thanks to Jacek Spera, @eoins, @pelger, @gbinside, @ManuEomm for reviews and suggestions.

loige.link/iter-sails - loige.link/async-it-code

for await (const _ of createAsyncCountdown(1_000_000)) {
  console.log("THANK YOU! 😍")
}

loige

nodejsdp.link

Finding a lost song with Node.js and async iterators

By Luciano Mammino

Finding a lost song with Node.js and async iterators

Did you ever get that feeling when a random song pops into your brain and you can’t get rid of it? Well, that happened to me recently and I couldn’t even remember the title of the damn song! In this talk, I want to share with you the story of how I was able to recover the details of the song by navigating some music-related APIs using JavaScript, Node.js and the magic of async iterators!

3,831

Luciano Mammino PRO

Cloud developer, entrepreneur, fighter, butterfly maker! #nodejs #javascript - Author of https://www.nodejsdesignpatterns.com , Founder of https://fullstackbulletin.com

Finding a lost song with Node.js & async iterators

Let me introduce myself first...

We are business focused technologists that deliver.

There was this song in my mind...

I could only remember some random parts and the word "dark" (probably in the title)

Let's give it a shot

It works! 🥳

Let's convert this to JavaScript

Seems good!

Let's look at the tracks...

The "sliding windows" problem 😩

Time based windows 😎

We have a working solution! 🎉 Can we generalise it?

We know how to iterate over every page/track. How do we expose this information?

How can we build an async iterator? 🧐

Meet the iteration protocols!

The iterator protocol

An object is an iterator if it has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value.

Generator functions "produce" iterators!

The iterable protocol

An object is iterable if it implements the @@iterator* method, a zero-argument function that returns an iterator.

OK. So far this is all synchronous iteration. What about async? 🙄

The async iterator protocol

An object is an async iterator if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value.

The async iterable protocol

An object is an async iterable if it implements the @@asyncIterator* method, a zero-argument function that returns an async iterator.

Now we know how to make our LastFmRecentTracks an Async Iterable 🤩

Let's search for all the songs that contain the word "dark" in their title! 🧐

For a more serious package that allows you to fetch data from Last.fm:

Finding a lost song with Node.js and async iterators

More from Luciano Mammino

Finding a lost song
with Node.js & async iterators

We have a working solution! 🎉
Can we generalise it?

We know how to iterate over every page/track.
How do we expose this information?

OK. So far this is all synchronous iteration.
What about async? 🙄