From Serializer to JsonEncoder

Robin Chalas

@bakslashHQ
@chalas_r

Mathias Arlaud

@bakslashHQ
@matarld

Representing data structures in a format that can be sent or persisted in order to be reconstructed later

Binary, textual

Construction pattern

Databases, flat files, APIs

Anywhere, interoperable

Serialization?

@chalas_r

@matarld

https://symfony.com/doc/current/components/serializer.html

Symfony's serializer

@chalas_r

@matarld

Symfony's serializer

{
  "name":"Axel",
  "color":"Red",
  "age":2
}

array(3) {
  ["name"]=> string(4) "Axel"
  ["color"]=> string(3) "Red"
  ["age"]=> int(2)
}

object(Cat)#1 (3) {
  ["name"]=> string(4) "Axel"
  ["color"]=> string(3) "Red"
  ["age"]=> int(2)
}

@chalas_r

@matarld

Some limitations

@chalas_r

@matarld

https://github.com/symfony/symfony/issues/30818

Design & BC limitations

@chalas_r

@matarld

AbstractNormalizer

allows attributes

instantiates objects

AbstractObjectNormalizer

normalization

denormalization

validation

ObjectNormalizer

extracts attributes

BC policy

🐌

Reflection is slow

Improved a lot over PHP versions, but still... Caching is needed.

The Reflection Limitation

@chalas_r

@matarld

The Groups Rabbit Hole

@chalas_r

@matarld

class Pet
{
    #[ORM\Column(type: 'boolean')]
    #[Groups([
        'cat_list:read',
        'cat:read',
        'cat:write'
    ])]
    private bool $meow = false;


    #[ORM\Column(type: 'boolean')]
    #[Groups([
        'dog_list:read',
        'dog:read',
        'dog:write'
    ])]
    private bool $bark = false;
}

@chalas_r

@matarld

class Cat
{
    private bool $meow = false;
}

class Dog
{
    private bool $bark = false;
}

1 Resource : 1 Represensation

@chalas_r

@matarld

class ListCat
{
    private string $name;
    private bool $meow;
}

class ReadCat
{
    private string $name;
    private float $height;
    private float $weight;
}

Metadata is data

Metadata

@chalas_r

@matarld

, but about data

data

metadata

"thing"

"animal"

"cat"

"flying cat"

@chalas_r

@matarld

Metadata

data

/** @temlate T of string */
final class Rain {
  /** @return list<T> */
  public function content(): array {...}
}

/** @var Rain<'cat'|'dog'> $*/
$englishRain = new Rain();

$fallingStuff = $englishRain->content();

type

no idea

array of something

list of something

list of cats and dogs

Possible JSON

[
  [0.7]
]

{
  "c": 1
}


true

{
  "a": 1
}

[
  3.14
]

[
  "pi"
]

[
  true
]

[
  3.14
]

[
  "dog"
]

[
  "cat"
]

[
  "dog"
]

[
  "dog"
]

The Type limitation

@chalas_r

@matarld

/**
 * @template T of Animal
 */
final class FlyingAnimal
{
  public function __construct(
    public object $type,
    private int $speed,
  ) {
  }
  
  /**
   * @return class-string<T>
   */
  public function getTypeClass(): string
  {
    return $this->type::class;
  }
  
  public function setSpeed(int $speed): void
  {
  	$this->speed = $speed;
  }
}

PropertyInfo

The Type limitation

@chalas_r

@matarld

TypeInfo component

https://github.com/symfony/symfony/pull/52510

@chalas_r

@matarld

/**
 * @template T of Animal
 */
final class FlyingAnimal
{
  public function __construct(
    public object $type,
    private int $speed,
  ) {
  }
  
  /**
   * @return class-string<T>
   */
  public function getTypeClass(): string
  {
    return $this->type::class;
  }
  
  public function setSpeed(int $speed): void
  {
  	$this->speed = $speed;
  }
}

TypeInfo

The Type limitation

@chalas_r

@matarld

Contribute to TypeInfo!

@chalas_r

@matarld

The Memory limitation

Huge collection of objects

Super big array

array

@chalas_r

@matarld

Json streaming

Huge collection of objects

array

@chalas_r

@matarld

JsonEncoder component

https://github.com/symfony/symfony/pull/51718

@chalas_r

@matarld

JsonEncoder component

PHP

metadata

Cat::class

class Cat
{
  public string $name;
  public bool $flying;
}

// ...

$this->jsonEncoder->encode($cat);

Possible JSON

{
  "name": "any_string",
  "flying: true|false
}

@chalas_r

@matarld

Encoder

Possible JSON

[  
  {
    "name": "any_string",
    "flying: true|false
  },

  {
    "name": "any_string",
    "flying: true|false
  }

return static function ($cats, $stream, $config) {
  $stream->write('[');
  $prefix_0 = '';
  foreach ($cats as $cat) {
    $stream->write($prefix_0);
    $stream->write('{"name":');
    $stream->write(json_encode($cat->name));
    $stream->write(',"flying":');
    $stream->write($cat->flying ? 'true' : 'false');
    $stream->write('}');
    $prefix_0 = ',';
  }
  $stream->write(']');
};

JsonEncoder component

@chalas_r

@matarld

I want to serialize a cat!

Does an encoder exist?

    /\_____/\
   /  o   o  \
  ( ==  ^  == )
   )         (
  (           )
 ( (  )   (  ) )
(__(__)___(__)__)

Encode the cat

Yep 👌

Compute the cat JSON shape

Generate and store the encoder

JsonEncoder component

@chalas_r

@matarld

Read only needed

JSON part lazily

JsonEncoder pros and cons

Blazing fast!

Flat memory usage

Stream ready

Generics ready

Simple API

Edit data on-the-fly

   __
o-''|\_____/)
 \_/|_)     )
    \  __  /
    (_/ (_/

@chalas_r

@matarld

@chalas_r

@matarld

composer req jolicode/automapper

Mapper

   __
o-''|\_____/)
 \_/|_)     )
    \  __  /
    (_/ (_/

AutoMapper

@chalas_r

@matarld

use AutoMapper\Attribute as Mapper;

final class Cat
{
  #[MapTo(
    target: Dog::class,
    property: 'barkVolume',
  )]
  public int $meowVolume;
}

final class Dog
{
  public int $barkVolume;
}

Renaming properties

barkVolume

meowVolume

@chalas_r

@matarld

use AutoMapper\Attribute as Mapper;

final class Cat
{
  #[MapTo(
    target: Dog::class,
    transformer: [self::class, 'toDogAge'],
  )]
  public int $age;
  
  public static function toDogAge(int $value, Source $source, array $context): int
  {
    return floor($value * 1.3);
  }
}

Updating values

@chalas_r

@matarld

final class Cat
{
  #[MapTo(
    target: Dog::class,
    transformer: CatToDogAgeTransformer::class,
  )]
  public int $age;
}

class CatToDogAgeTransformer implements PropertyTransformerInterface
{
  public function __construct(
    private AgeConverter $ageConverter,
  ) {}

  public function transform(int $value, object|array $source, array $context): mixed
  {
    return $this->ageConverter->fromCatToDog($value);
  }
}

Updating values

use Symfony\Component\Serializer\SerializerInterface;

final readonly class MyService
{
  public function __construct(
    private SerializerInterface $serializer,
  ) {
  }
}

Migration level 0-1

@chalas_r

@matarld

// config/bundles.php

return [
  // ...
  Mtarld\JsonEncoderBundle\JsonEncoderBundle::class => ['all' => true],
  AutoMapper\Bundle\AutoMapperBundle::class => ['all' => true],
  TurboSerializer\TurboSerializerBundle::class => ['all' => true],
];

composer require korbeil/turbo-serializer

Migration level 0-1

@chalas_r

@matarld

use Symfony\Component\Serializer\SerializerInterface;
use TurboSerializer\Serializer;

final readonly class ApiController
{
  public function __construct(
    private SerializerInterface $turboSerializer,
  ) {
  }

  #[Route('/api/cats')]
  public function get(): Response
  {
    $cats = $this->getCats();
    $type = Type::iterable(Type::object(Cat::class), Type::int(), asList: true);
    
    return new Response($this->turboSerializer->serialize(
      $cats, 
      'json',
      [Serializer::TYPE => $type]
    ));
  }
}

Serialization level 0-1

@chalas_r

@matarld

final readonly class ApiController
{
  public function __construct(
    private SerializerInterface $turboSerializer,
  ) {
  }

  #[Route('/api/create-cats')]
  public function post(string $content): Response
  {
    $cats = $this->turboSerializer->deserialize($content, Cat::class.'[]', 'json');
  }
}

Deserialization level 0-1

@chalas_r

@matarld

final readonly class ApiController
{
  public function __construct(
    private SerializerInterface $turboSerializer,
  ) {
  }

  #[Route('/api/cats-but-dogs')]
  public function get(): Response
  {
    $cats = $this->getCats();
    $type = Type::iterable(Type::object(Cat::class), Type::int(), asList: true);
    
    return new Response($this->turboSerializer->serialize(
      $cats, 
      'json',
      [Serializer::NORMALIZED_TYPE => $type]
    ));
  }
}

Serialization level 0-2

@chalas_r

@matarld

final readonly class ApiController
{
	public function __construct(
    	private SerializerInterface $turboSerializer,
    ) {
    }

    #[Route('/api/create-cats')]
    public function post(string $content): Response
    {
        $dogsType = Type::iterable(Type::object(Dog::class), Type::int(), asList: true);
    
        $this->turboSerializer->deserialize(
      	    $content, 
	        Cat::class.[], 
    	    'json', 
	        [Serializer::NORMALIZED_TYPE => $dogsType],    
        );

	    // ...
    }
}

Deserialization level 0-2

@chalas_r

@matarld

final readonly class ApiController
{
  public function __construct(
    private EncoderInterface $jsonEncoder,
  ) {
  }

  #[Route('/api/cats')]
  public function get(): Response
  {
    $cats = $this->getCats();
    $type = Type::iterable(Type::object(Cat::class), Type::int(), asList: true);
    
    return new Response((string) $this->jsonEncoder->encode($cats, $type));
  }
}

Serialization level 1-1

@chalas_r

@matarld

final readonly class ApiController
{
  public function __construct(
    private DecoderInterface $jsonDecoder,
  ) {
  }

  #[Route('/api/create-cats')]
  public function post(string $content): Response
  {
    $type = Type::iterable(Type::object(Cat::class), Type::int(), asList: true);
    $cats = $this->jsonDecoder->decode($content, $type);
  }
}

Deserialization level 1-1

@chalas_r

@matarld

final readonly class ApiController
{
  public function __construct(
    private EncoderInterface $jsonEncoder,
  ) {
  }

  #[Route('/api/cats')]
  public function get(): Response
  {
    $cats = $this->getCats();
    $type = Type::iterable(Type::object(Cat::class), Type::int(), asList: true);
    
    return new StreamedResponse(function () use ($cats, $type): void {
      foreach ($this->jsonEncoder->encode($cats, $type) as $chunk) {
        echo $chunk;
      }
    });
  }
}

Serialization level 1-2

@chalas_r

@matarld

final readonly class ApiController
{
  public function __construct(
    private EncoderInterface $jsonEncoder,
  ) {
  }

  #[Route('/api/cats')]
  public function get(): Response
  {
    $cats = $this->getCats();
    $type = Type::iterable(Type::object(Cat::class), Type::int(), asList: true);
    
    $this->jsonEncoder->encode($cats, $type, ['stream' => new OutputStream()]);
    
    return new Response();
  }
}

Serialization level 1-3

@chalas_r

@matarld

final readonly class ApiController
{
  public function __construct(
    private DecoderInterface $jsonDecoder,
  ) {
  }

  #[Route('/api/create-cats')]
  public function post(): Response
  {
    $type = Type::iterable(Type::object(Cat::class), Type::int(), asList: true);
    $cats = $this->jsonDecoder->decode(new InputStream(), $type);
    foreach ($cats as $i => $cat) {
      if ($i === 1000) {
        dd($cat->name);
      }
    }
  }
}

Deserialization level 1-3

@chalas_r

@matarld

final readonly class ApiController
{
  public function __construct(
    private EncoderInterface $jsonEncoder,
    private AutoMapperInterface $autoMapper,
  ) {
  }

  #[Route('/api/cats-but-dogs')]
  public function get(): Response
  {
    $cats = $this->getCats();
    $type = Type::iterable(Type::object(Dog::class), Type::int(), asList: true);
    $this->jsonEncoder->encode($this->catsToDogs($cats), $type, ['stream' => new OutputStream()]);
    
    return new Response();
  }
  
  private function catsToDogs(iterable $cats): iterable
  {
    foreach ($cats as $cat) {
      yield $this->autoMapper->map($cat, Dog::class);
    }
  }
}

Serialization level 1-4

@chalas_r

@matarld

final readonly class ApiController
{
  public function __construct(
    private DecoderInterface $jsonDecoder,
    private AutomapperInterface $mapper,
  ) {
  }

  #[Route('/api/create-cats')]
  public function post(): Response
  {
    $type = Type::iterable(Type::object(Cat::class), Type::int(), asList: true);
    $this->catsToDogs($this->jsonDecoder->decode(new InputStream(), $type));
  }
    
  private function catsToDogs(iterable $cats): iterable
  {
    foreach ($cats as $cat) {
      yield $this->autoMapper->map($cat, Dog::class);
    }
  }
}

Deserialization level 1-4

@chalas_r

@matarld

Benchmark

Deserialization

Serializer

JsonEncoder

JsonEncoder
+ Automapper

Serialization

JsonEncoder

Serializer

JsonEncoder
+ Automapper

@chalas_r

@matarld

Thanks!

https://github.com/mtarld/json-encoder-bundle

https://github.com/jolicode/automapper

https://github.com/korbeil/turbo-serializer

https://github.com/korbeil/turbo-serializer-bench

Robin Chalas

@bakslashHQ
@chalas_r

Mathias Arlaud

@bakslashHQ
@matarld

From Serializer to JsonEncoder

Serialization?

Symfony's serializer

Symfony's serializer

Some limitations

Design & BC limitations

🐌

The Reflection Limitation

The Groups Rabbit Hole

1 Resource : 1 Represensation

1 Resource : 1 Represensation

Metadata

Metadata

The Type limitation

The Type limitation

TypeInfo component

The Type limitation

Contribute to TypeInfo!

The Memory limitation

Json streaming

JsonEncoder component

JsonEncoder component

JsonEncoder component

JsonEncoder component

JsonEncoder pros and cons

Mapper

AutoMapper

Renaming properties

Updating values

Updating values

Migration level 0-1

Migration level 0-1

Serialization level 0-1

Deserialization level 0-1

Serialization level 0-2

Deserialization level 0-2

Serialization level 1-1

Deserialization level 1-1

Serialization level 1-2

Serialization level 1-3

Deserialization level 1-3

Serialization level 1-4

Deserialization level 1-4

Benchmark

Thanks!

[Meetup] From Serializer to JsonEncoder

More from Mathias Arlaud