Did you Know
JavaScript has Iterators?

Luciano Mammino (@loige)

loige.link/iterate

2023-03-07

META_SLIDE!

loige

loige.link/iterate

loige

String[] transactions = {
  "paid 20",
  "received 10",
  "paid 5",
  "received 15",
  "paid 10",
  "received 12"
};

var total = Stream.of(transactions).mapToInt(transaction -> {
  var parts = transaction.split(" ");
  int amount = Integer.decode(parts[1]);
  if (Objects.equals(parts[0], "paid")) {
    amount = -amount;
  }

  return amount;
}).sum();

if (total >= 0) {
  System.out.println("Life is good :)");
} else {
  System.out.println("You are broke :(");
}

replit.com/@lmammino/r-u-broke-java

loige

String[] transactions = {
  "paid 20",
  "received 10",
  "paid 5",
  "received 15",
  "paid 10",
  "received 12"
};

var total = Stream.of(transactions).mapToInt(transaction -> {
  var parts = transaction.split(" ");
  int amount = Integer.decode(parts[1]);
  if (Objects.equals(parts[0], "paid")) {
    amount = -amount;
  }

  return amount;
}).sum();

if (total >= 0) {
  System.out.println("Life is good :)");
} else {
  System.out.println("You are broke :(");
}

replit.com/@lmammino/r-u-broke-java

loige

transactions = [
  "paid 20",
  "received 10",
  "paid 5",
  "received 15",
  "paid 10",
  "received 12"
];

def get_amount(transaction):
  type, amount = transaction.split(' ')
  amount = int(amount)
  if type == 'paid':
    return -amount
  return amount

*_, total = accumulate(
  map(get_amount, transactions)
)

if total >= 0:
  print("Life is good :)")
else:
  print("You are broke :(")

replit.com/@lmammino/r-u-broke-py

loige

let transactions = [
  "paid 20",
  "received 10",
  "paid 5",
  "received 15",
  "paid 10",
  "received 12",
];

let total: i32 = transactions
  .iter()
  .map(|transaction| {
    let (action, amount) = transaction.split_once(' ').unwrap();
    let amount = amount.parse::<i32>().unwrap();
    if action == "paid" {
      -amount
    } else {
      amount
    }
  })
  .sum();

if total >= 0 {
  println!("Life is good :)");
} else {
  println!("You are broke :(");
}

replit.com/@lmammino/r-u-broke-rust

loige

const transactions = [
  "paid 20",
  "received 10",
  "paid 5",
  "received 15",
  "paid 10",
  "received 12"
]

const total = Iterator.from(transactions)
.map(transaction => {
  let [action, amount] = transaction.split(" ")
  amount = Number.parseInt(amount)
  if (action === "paid") {
    return -amount
  } else {
    return amount
  }
})
.reduce((acc,curr) => acc + curr)

if (total >= 0) {
  console.log("Life is good :)")
} else {
  console.log("You are broke :(");
}

replit.com/@lmammino/r-u-broke-js

loige

const transactions = [
  "paid 20",
  "received 10",
  "paid 5",
  "received 15",
  "paid 10",
  "received 12"
]

const total = Iterator.from(transactions)
.map(transaction => {
  let [action, amount] = transaction.split(" ")
  amount = Number.parseInt(amount)
  if (action === "paid") {
    return -amount
  } else {
    return amount
  }
})
.reduce((acc,curr) => acc + curr)

if (total >= 0) {
  console.log("Life is good :)")
} else {
  console.log("You are broke :(");
}

replit.com/@lmammino/r-u-broke-js

loige

const transactions = [
  "paid 20",
  "received 10",
  "paid 5",
  "received 15",
  "paid 10",
  "received 12"
]

const total = Iterator.from(transactions)
.map(transaction => {
  let [action, amount] = transaction.split(" ")
  amount = Number.parseInt(amount)
  if (action === "paid") {
    return -amount
  } else {
    return amount
  }
})
.reduce((acc,curr) => acc + curr)

if (total >= 0) {
  console.log("Life is good :)")
} else {
  console.log("You are broke :(");
}

replit.com/@lmammino/r-u-broke-js

This is not JavaScript, it's FutureJavaScript™️

github.com/tc39/proposal-iterator-helpers

loige

import Iterator from 'core-js-pure/actual/iterator/index.js'

const transactions = [
  "paid 20",
  "received 10",
  "paid 5",
  "received 15",
  "paid 10",
  "received 12"
]

const total = Iterator.from(transactions)
.map(transaction => {
  let [action, amount] = transaction.split(" ")
  amount = Number.parseInt(amount)
  if (action === "paid") {
    return -amount
  } else {
    return amount
  }
})
.reduce((acc,curr) => acc + curr)

if (total >= 0) {
  console.log("Life is good :)")
} else {
  console.log("You are broke :(");
}

But if you want the future, today...

npm i --save core-js-pure

loige

WHY? 🤷‍♀️

loige

Iterators are lazy!

  • You can consume the collection 1 item at the time
  • You don't need to keep all the items in memory
  • Great for large datasets
  • You can even have endless iterators!

loige

The concept of Iterators exists already in JavaScript...

🗞️ Good news FROM THE WORLD...

... since ES2015!

WAIT, Who the heck is this guy !? 🤷

👋 I'm Luciano (🇮🇹🍕🍝🤌)

👨‍💻 Senior Architect @ fourTheorem (Dublin 🇮🇪)

nodejsdp.link

📔 Co-Author of Node.js Design Patterns  👉

Let's connect!

  loige.co (blog)

  @loige (twitter)

  loige (twitch)

  lmammino (github)

Always re-imagining

We are a pioneering technology consultancy focused on Cloud, AWS & serverless

loige

 

😇 We are always looking for talent: fth.link/careers

We can help with:

Cloud Migrations

Training & Cloud enablement

Building serverless applications

Cutting cloud costs

loige

awsbites.com podcast

If you like AWS...

loige

🥤

Let's have a quick refreshER
on some JS "Iteration" concepts...

loige

const array = ['foo', 'bar', 'baz']

for (const item of array) {
  console.log(item)
}

 

Prints all the items in the array!

Does it need to be an array? 🤔

Output:
foo
bar
baz

loige

const str = 'foo'

for (const item of str) {
  console.log(item)
}
Output:
f
o
o

loige

const set = new Set(['foo', 'bar', 'baz'])

for (const item of set) {
  console.log(item)
}
Output:
foo
bar
baz

loige

const map = new Map([
  ['foo', 'bar'], ['baz', 'qux']
])

for (const item of map) {
  console.log(item)
}
Output:
[ 'foo', 'bar' ]
[ 'baz', 'qux' ]

loige

const obj = {
  foo: 'bar',
  baz: 'qux'
}

for (const item of obj) {
  console.log(item)
}
Output:
⛔️ Uncaught TypeError: obj is not iterable

OMG `for ... of`

does not work with plain objects! 😱

loige

const obj = {
  foo: 'bar',
  baz: 'qux'
}

for (const item of Object.entries(obj)) {
  console.log(item)
}
Output:
[ 'foo', 'bar' ]
[ 'baz', 'qux' ]

loige

loige

const array = ['foo', 'bar', 'baz']

console.log(...array)
Output:
foo bar baz

spread syntax!

loige

loige

📒 AGENDA

  • Generators
  • Iterator protocol
  • Iterable protocol
  • Async Iterator protocol
  • Async Iterable protcol
  • Real-life™️ examples

loige

➡️ Generators

loige

Generator fn & obj

loige

function * myGenerator () {
  // generator body
  yield 'someValue'
  // ... do more stuff
}
const genObj = myGenerator()
genObj.next() // -> { done: false, value: 'someValue' }

loige

function * fruitGen () {
  yield '🍑'
  yield '🍉'
  yield '🍋'
  yield '🥭'
}

const fruitGenObj = fruitGen()
console.log(fruitGenObj.next()) // { value: '🍑', done: false }
console.log(fruitGenObj.next()) // { value: '🍉', done: false }
console.log(fruitGenObj.next()) // { value: '🍋', done: false }
console.log(fruitGenObj.next()) // { value: '🥭', done: false }
console.log(fruitGenObj.next()) // { value: undefined, done: true }

loige

function * fruitGen () {
  yield '🍑'
  yield '🍉'
  yield '🍋'
  yield '🥭'
}

const fruitGenObj = fruitGen()
// generator objects are iterable!
for (const fruit of fruitGenObj) {
  console.log(fruit)
}

// 🍑
// 🍉
// 🍋
// 🥭

loige

function * range (start, end) {
  for (let i = start; i < end; i++) {
    yield i
  }
}

// generators are lazy!
for (const i of range(0, Number.MAX_VALUE)) {
  console.log(i)
}

const zeroToTen = [...range(0, 11)]

loige

// generators can be "endless"
function * cycle (values) {
  let current = 0
  while (true) {
    yield values[current]
    current = (current + 1) % values.length
  }
}

for (const value of cycle(['even', 'odd'])) {
  console.log(value)
}

// even
// odd
// even
// ...

📝 Mini-summary

loige

  • A generator function returns a generator object which is both an iterator and an iterable.
  • A generator function uses `yield` to yield a value and pause its execution. The generator object is used to make progress on an instance of the generator (by calling `next()`).
  • Generator functions are a great way to create custom iterable objects.
  • Generator objects are lazy and they can be endless.

➡️ Iterators & Iterables

loige

Iterator Obj

loige

An object that acts like a cursor to iterate over blocks of data sequentially

Iterable Obj

loige

An object that contains data that can be iterated over sequentially

The iterator protocol

An object is an iterator if it has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value.

loige

function createCountdown (from) {
  let nextVal = from
  return {
    next () {
      if (nextVal < 0) {
        return { done: true }
      }

      return { 
        done: false,
        value: nextVal--
      }
    }
  }
}

loige

A factory function that creates an iterator

returns an object

... which has a next() method

... which returns an object with

done & value

const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }

console.log(countdown.next())
// { done: false, value: 2 }

console.log(countdown.next())
// { done: false, value: 1 }

console.log(countdown.next())
// { done: false, value: 0 }

console.log(countdown.next())
// { done: true }

loige

🔥
Generator Objects are iterators (and iterables)!

loige

function * createCountdown (from) {
  for (let i = from; i >= 0; i--) {
    yield i
  }
}

loige

const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }

console.log(countdown.next())
// { done: false, value: 2 }

console.log(countdown.next())
// { done: false, value: 1 }

console.log(countdown.next())
// { done: false, value: 0 }

console.log(countdown.next())
// { done: true, value: undefined }

loige

The iterable protocol

An object is iterable if it implements the Symbol.iterator method, a zero-argument function that returns an iterator.

loige

function createCountdown (from) {
  let nextVal = from
  return {
    [Symbol.iterator]: () => ({
      next () {
        if (nextVal < 0) {
          return { done: true }
        }

        return { done: false, value: nextVal-- }
      }
    })
  }
}

loige

function createCountdown (from) {
  return {
    [Symbol.iterator]: function * () {
      for (let i = from; i >= 0; i--) {
        yield i
      }
    }
  }
}

loige

function * createCountdown () {
  for (let i = from; i >= 0; i--) {
    yield i
  }
}

loige

🔥 or just use generators!

const countdown = createCountdown(3)

for (const value of countdown) {
  console.log(value)
}

// 3
// 2
// 1
// 0

loige

An object can be an iterable and an iterator at the same time!

const iterableIterator = {
  next () {
    return { done: false, value: 'hello' }
  },
  [Symbol.iterator] () {
    return this
  }
}

loige

📝 Mini-summary 1/2

loige

  • An iterator is an object that allows us to traverse a collection

  • The iterator protocol specifies that an object is an iterator if it has a `next()` method that returns an object with the shape `{done, value}`.

    • `done` (a boolean) tells us if the iteration is completed

    • `value` represents the value from the current iteration.

  • You can write an iterator as an anonymous object (e.g. returned by a factory function), using classes or using generators.

📝 Mini-summary 2/2

loige

  • The iterable protocol defines what's expected for a JavaScript object to be considered iterable. That is an object that holds a collection of data on which you can iterate on sequentially.

  • An object is iterable if it implements a special method called `Symbol.iterator` which returns an iterator. (An object is iterable if you can get an iterator from it!)

  • Generator functions produce objects that are iterable.

  • We saw that generators produce objects that are also iterators.

  • It is possible to have objects that are both iterator and iterable. The trick is to create the object as an iterator and to implement a `Symbol.iterator` that returns the object itself (`this`).

OK, very cool!
But, so far this is all synchronous iteration.
What about async? 🙄

loige

➡️ ASYNC Iterators & Iterables

loige

The async iterator protocol

An object is an async iterator if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value.

loige

import { setTimeout } from 'node:timers/promises'

function createAsyncCountdown (from, delay = 1000) {
  let nextVal = from
  return {
    async next () {
      await setTimeout(delay)
      if (nextVal < 0) {
        return { done: true }
      }

      return { done: false, value: nextVal-- }
    }
  }
}

loige

const countdown = createAsyncCountdown(3)
console.log(await countdown.next())
// { done: false, value: 3 }

console.log(await countdown.next())
// { done: false, value: 2 }

console.log(await countdown.next())
// { done: false, value: 1 }

console.log(await countdown.next())
// { done: false, value: 0 }

console.log(await countdown.next())
// { done: true }

loige

loige

import { setTimeout } from 'node:timers/promises'

// async generators "produce" async iterators!

async function * createAsyncCountdown (from, delay = 1000) {
  for (let i = from; i >= 0; i--) {
    await setTimeout(delay)
    yield i
  }
}

loige

The async iterable protocol

An object is an async iterable if it implements the `Symbol.asyncIterator` method, a zero-argument function that returns an async iterator.

loige

import { setTimeout } from 'node:timers/promises'

function createAsyncCountdown (from, delay = 1000) {
  return {
    [Symbol.asyncIterator]: async function * () {
      for (let i = from; i >= 0; i--) {
        await setTimeout(delay)
        yield i
      }
    }
  }
}

loige

HOT TIP 🔥

With async generators we can create objects that are both async iterators and async iterables!

(We don't need to specify Symbol.asyncIterator explicitly!)

loige

import { setTimeout } from 'node:timers/promises'

// async generators "produce" async iterators
// (and iterables!)

async function * createAsyncCountdown (from, delay = 1000) {
  for (let i = from; i >= 0; i--) {
    await setTimeout(delay)
    yield i
  }
}

loige

const countdown = createAsyncCountdown(3)

for await (const value of countdown) {
  console.log(value)
}

loige

📝 Mini-summary 1/2

loige

  • Async iterators are the asynchronous counterpart of iterators.

  • They are useful to iterate over data that becomes available asynchronously (e.g. coming from a database or a REST API).

  • A good example is a paginated API, we could build an async iterator that gives a new page for every iteration.

  • An object is an async iterator if it has a `next()` method which returns a `Promise` that resolves to an object with the shape: `{done, value}`.

  • The main difference with the iterator protocol is that this time `next()` returns a promise.

  • When we call next we need to make sure we `await` the returned promise.

📝 Mini-summary 2/2

loige

  • The async iterable protocol defines what it means for an object to be an async iterable.

  • Once you have an async iterable you can use the `for await ... of` syntax on it.

  • An object is an async iterable if it has a special method called `Symbol.asyncIterator` that returns an async iterator.

  • Async iterables are a great way to abstract paginated data that is available asynchronously or similar operations like pulling jobs from a remote queue.

  • A small spoiler, async iterables can also be used with Node.js streams...

A reaL USE CASE & A CHALLENGE!

loige

Async Iterables are commonly used to abstract over paginated resources. Can you implement a paginator for the Rick & Morty API?

function createCharactersPaginator () {
  // return an async iterable object that produces 
  // pages with name of Rick and Morty characters
  // taken from the API
  // https://rickandmortyapi.com/api/character
}

// This is what we want to support 👇
const paginator = createCharactersPaginator()
for await (const page of paginator) {
  console.log(page)
}

In conclusion 💥

loige

  • JavaScript has built-in sync/async iteratable/iterators
  • These are great tools when you need to deal with large (or even endless) data sets
  • They give you great control over how to consume data incrementally
  • Iterator helpers are still lacking and not as mature as in other languages
  • ... although, you should know enough to be able to create your own helpers now 😜
  • ... and you can leverage generator functions to make your life easier!
  • ... or you can wait for Iterator Helpers & Async Iterator Helpers to land
  • ... or you can use core-js to polyfill these features today!

BONUS: a free workshop for you! 🎁

loige

loige.link/lets-iter-repo

git clone https://github.com/lmammino/iteration-protocols-workshop.git
cd iteration-protocols-workshop
npm i

Front cover Photo by Sam Barber on Unsplash

Back cover photo by photonblast on Unsplash

fourtheorem.com

THANKS! 🙌

loige

nodejsdp.link

loige.link/iterate

Did you know JavaScript has iterators? DublinJS

By Luciano Mammino

Did you know JavaScript has iterators? DublinJS

How many ways do you know to do iteration with JavaScript and Node.js? While, for loop, for…in, for..of, .map(), .forEach(), streams, iterators, etc! Yes, there are a lot of ways! But did you know that JavaScript has iteration protocols to standardise synchronous and even asynchronous iteration? In this workshop we will learn about these protocols and discover how to build iterators and iterable objects, both synchronous and asynchronous. We will learn about some common use cases for these protocols, explore generators and async generators (great tools for iteration) and finally discuss some hot tips, common pitfalls, and some (more or less successful) wild ideas!

  • 2,254

More from Luciano Mammino