WIP: DB Modules Blog #26

dfellis · 2022-04-13T04:50:51Z

aguillenv · 2022-04-13T12:29:01Z

blog/2022-04-11-db-modules.mdx

+
+* Reduced the amount of tooling necessary to learn and set up.
+* Increased the scenarios in which managing your IaSQL database is possible.
+* Made the results of the operations themsevles queryable 


Suggested change

* Made the results of the operations themsevles queryable

* Made the results of the operations themselves queryable.

aguillenv · 2022-04-13T12:30:20Z

blog/2022-04-11-db-modules.mdx

+
+Managing your cloud infrastructure with the many ways they can be configured and the interlocking relationships between them maps very cleanly into a database infrastructure. The various cloud entities become rows in tables, the configuration options become columns, the relationships between them represented by joins between tables using foreign keys, and invalid configurations can be prevented in the first place through constraints. This model of the cloud also enables the importing of existing infrastructure into the database, allowing one to go from un-managed to managed infrastructure in seconds.
+
+But we also recognized that we could not produce an accurate model of the entirety of AWS's offerings in one shot, and that we might make mistakes in our modeling, and so focused on an initial subset of functionality with the intention of building it up and fixing it as we encounter edge cases. This is already a solved problem in databases with schema migration systems, or so we thought. But as we built our first prototype, we realized that our prototype was becoming unweildy and off-putting even to us as its creators. It was becoming harder to figure out what you wanted to do to manipulate your cloud and execution time was slowing down. The root cause of both of these was the database schema.


Looked up for the word and I thing there's a typo:

Suggested change

But we also recognized that we could not produce an accurate model of the entirety of AWS's offerings in one shot, and that we might make mistakes in our modeling, and so focused on an initial subset of functionality with the intention of building it up and fixing it as we encounter edge cases. This is already a solved problem in databases with schema migration systems, or so we thought. But as we built our first prototype, we realized that our prototype was becoming unweildy and off-putting even to us as its creators. It was becoming harder to figure out what you wanted to do to manipulate your cloud and execution time was slowing down. The root cause of both of these was the database schema.

But we also recognized that we could not produce an accurate model of the entirety of AWS's offerings in one shot, and that we might make mistakes in our modeling, and so focused on an initial subset of functionality with the intention of building it up and fixing it as we encounter edge cases. This is already a solved problem in databases with schema migration systems, or so we thought. But as we built our first prototype, we realized that our prototype was becoming unwieldy and off-putting even to us as its creators. It was becoming harder to figure out what you wanted to do to manipulate your cloud and execution time was slowing down. The root cause of both of these was the database schema.

aguillenv · 2022-04-13T12:33:12Z

blog/2022-04-11-db-modules.mdx

+Finally, while having each module scope its tables into its own schema namespace would eliminate the possibility of collisions, [the majority of ORMs and migration systems do not support multiple schemas simultaneously](https://github.com/prisma/prisma/issues/1122), which would make adoption more difficult. With a metadata table associating table names to module/package names, the difficulties of knowing what tables belong to which is also somewhat mitigated.
+


I think we should add here explicitly the option we are choosing even if it can be tell by discard?

aguillenv · 2022-04-13T12:33:35Z

blog/2022-04-11-db-modules.mdx

+
+This can be used for the immutable metadata tables, too. By isolating them into a separate module that depends on the "main" module, you not only make it clearer to the user that they're separate and immutable (by the module name), you can bolt on increased type safety by changing a column into a foreign key on one of the metadata tables when it is installed and reverting it back if uninstalled. Now instead of an EC2 instance's AMI ID being just an opaque string, it can be a string that's joined onto the AMI ID of an AMI table in an `ec2_metadata` module and you can be sure you've got no typos when you try to deploy because the database wouldn't even let you insert the record in the first place if it was wrong.
+
+## ["So Help Me Cobb"](https://en.wikipedia.org/wiki/Third_normal_form#cite_note-Diehr-8)


Isn't it:

Suggested change

## ["So Help Me Cobb"](https://en.wikipedia.org/wiki/Third_normal_form#cite_note-Diehr-8)

## ["So Help Me Codd"](https://en.wikipedia.org/wiki/Third_normal_form#cite_note-Diehr-8)

aguillenv · 2022-04-13T12:36:03Z

blog/2022-04-11-db-modules.mdx

+
+## ["So Help Me Cobb"](https://en.wikipedia.org/wiki/Third_normal_form#cite_note-Diehr-8)
+
+On sticking to third normal form, we realized that if we loosened up our ORM target a bit to only the most popular ORMs, we can use Postgres Arrays and ActiveRecord, Django, TypeORM, and Prisma, amongst others all support this feature, and so for fields that truly have no place outside of a singular entity and truly should always be created, updated, and deleted in sync, we can switch to that, reducing the number of tables necessary.


I found this paragraph a bit confusing. Maybe something like:

Suggested change

On sticking to third normal form, we realized that if we loosened up our ORM target a bit to only the most popular ORMs, we can use Postgres Arrays and ActiveRecord, Django, TypeORM, and Prisma, amongst others all support this feature, and so for fields that truly have no place outside of a singular entity and truly should always be created, updated, and deleted in sync, we can switch to that, reducing the number of tables necessary.

On sticking to third normal form, we realized that if we loosened up our ORM target a bit to only the most popular ORMs (Django, TypeORM, and Prisma, amongst others), we can use Postgres Arrays and ActiveRecord for fields that truly have no place outside of a singular entity and truly should always be created, updated, and deleted in sync, we can switch to that, reducing the number of tables necessary.

WIP: DB Modules Blog

6e4e46a

dfellis self-assigned this Apr 13, 2022

dfellis requested review from depombo and aguillenv April 13, 2022 04:50

aguillenv reviewed Apr 13, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: DB Modules Blog #26

WIP: DB Modules Blog #26

dfellis commented Apr 13, 2022 •

edited

Loading

aguillenv Apr 13, 2022

aguillenv Apr 13, 2022

aguillenv Apr 13, 2022

aguillenv Apr 13, 2022

aguillenv Apr 13, 2022

	* Made the results of the operations themsevles queryable
	* Made the results of the operations themselves queryable.


		Managing your cloud infrastructure with the many ways they can be configured and the interlocking relationships between them maps very cleanly into a database infrastructure. The various cloud entities become rows in tables, the configuration options become columns, the relationships between them represented by joins between tables using foreign keys, and invalid configurations can be prevented in the first place through constraints. This model of the cloud also enables the importing of existing infrastructure into the database, allowing one to go from un-managed to managed infrastructure in seconds.

		But we also recognized that we could not produce an accurate model of the entirety of AWS's offerings in one shot, and that we might make mistakes in our modeling, and so focused on an initial subset of functionality with the intention of building it up and fixing it as we encounter edge cases. This is already a solved problem in databases with schema migration systems, or so we thought. But as we built our first prototype, we realized that our prototype was becoming unweildy and off-putting even to us as its creators. It was becoming harder to figure out what you wanted to do to manipulate your cloud and execution time was slowing down. The root cause of both of these was the database schema.

		Finally, while having each module scope its tables into its own schema namespace would eliminate the possibility of collisions, [the majority of ORMs and migration systems do not support multiple schemas simultaneously](https://github.com/prisma/prisma/issues/1122), which would make adoption more difficult. With a metadata table associating table names to module/package names, the difficulties of knowing what tables belong to which is also somewhat mitigated.


		This can be used for the immutable metadata tables, too. By isolating them into a separate module that depends on the "main" module, you not only make it clearer to the user that they're separate and immutable (by the module name), you can bolt on increased type safety by changing a column into a foreign key on one of the metadata tables when it is installed and reverting it back if uninstalled. Now instead of an EC2 instance's AMI ID being just an opaque string, it can be a string that's joined onto the AMI ID of an AMI table in an `ec2_metadata` module and you can be sure you've got no typos when you try to deploy because the database wouldn't even let you insert the record in the first place if it was wrong.

		## ["So Help Me Cobb"](https://en.wikipedia.org/wiki/Third_normal_form#cite_note-Diehr-8)

	## ["So Help Me Cobb"](https://en.wikipedia.org/wiki/Third_normal_form#cite_note-Diehr-8)
	## ["So Help Me Codd"](https://en.wikipedia.org/wiki/Third_normal_form#cite_note-Diehr-8)


		## ["So Help Me Cobb"](https://en.wikipedia.org/wiki/Third_normal_form#cite_note-Diehr-8)

		On sticking to third normal form, we realized that if we loosened up our ORM target a bit to only the most popular ORMs, we can use Postgres Arrays and ActiveRecord, Django, TypeORM, and Prisma, amongst others all support this feature, and so for fields that truly have no place outside of a singular entity and truly should always be created, updated, and deleted in sync, we can switch to that, reducing the number of tables necessary.

WIP: DB Modules Blog #26

Are you sure you want to change the base?

WIP: DB Modules Blog #26

Conversation

dfellis commented Apr 13, 2022 • edited Loading

aguillenv Apr 13, 2022

Choose a reason for hiding this comment

aguillenv Apr 13, 2022

Choose a reason for hiding this comment

aguillenv Apr 13, 2022

Choose a reason for hiding this comment

aguillenv Apr 13, 2022

Choose a reason for hiding this comment

aguillenv Apr 13, 2022

Choose a reason for hiding this comment

dfellis commented Apr 13, 2022 •

edited

Loading