As the volume of data continues to expand rapidly, so does the demand for efficient data management and observability solutions and tools. The actual value of data lies in how it is being utilized. Collecting and storing data alone is not enough; it must be leveraged and used correctly to get valuable insights. These insights can range from demographics to consumer behavior and even future sales predictions, providing an unparalleled resource for business decision-making processes. Also, with real-time data, businesses can make quick and informed decisions, adapt to the market and capitalize on live opportunities. However, this is only possible if data is of good quality, outdated, misleading, or difficult to access, which is precisely where DataOps comes to the rescue and plays a crucial role in optimizing and streamlining data management processes.

Unpacking the essence of DataOps

DataOps is a set of best practices and tools that aims to enhance the collaboration, integration, and automation of data management operations and tasks. DataOps seeks to improve the quality, speed, and collaboration of data management through an integrated and process-oriented approach, utilizing automation and agile software engineering practices similar to that of DevOps to speed up and streamline the process of accurate data delivery [1]. It is designed to help businesses and organizations better manage their data pipelines, reduce the workload and time required to develop and deploy new data-driven applications and improve the quality of the data being used.

Now that we have a clear understanding of what DataOps means let's delve deeper into its key components. The key components of a DataOps strategy include data integration, data quality management and measurement, data governance, data orchestration, and DataOps Observability.

Data integration

Data integration involves integrating and testing code changes and promptly deploying them to production environments, ensuring accuracy and consistency of data as it is integrated and delivered to appropriate teams.

Data quality management

Data Quality Management involves identifying, correcting, and preventing errors or inconsistencies in data, ensuring that the data being used is highly reliable and accurate.

Data governance

Data governance ensures that data is collected, stored, and used consistently, ethically and complies with regulations.

Data orchestration

Data orchestration helps to manage and coordinates data processing in a pipeline, specifying and scheduling tasks and dealing with errors to automate and optimize data flow through the data pipeline. It is crucial for ensuring the smooth operation and performance of the data through the data pipeline.

DataOps observability

DataOps observability refers to the ability to monitor and understand the various processes and systems involved in data management, with the primary goal of ensuring the reliability, trustworthiness, and business value of the data. It involves everything from monitoring and analyzing data pipelines to maintaining data quality and proving the data's business value through financial and operational efficiency metrics. DataOps observability allows businesses and organizations to improve the efficiency of their data management processes and make better use of their data assets. It aids in ensuring that data is always correct, dependable, and easily accessible, which in turn helps businesses and organizations make data-driven decisions, optimize data-related costs/spend and generate more value from it.

Top DataOps and DataOps Observability tools to simplify data management, cost & collaboration processes

One of the most challenging aspects of DataOps is integrating data from various sources and ensuring data quality, orchestration, observability, data cost management, and governance. DataOps aims to streamline these processes and improve collaboration among teams, enabling businesses to make better data-driven decisions and achieve improved performance and results [2]. In this article, we will focus on DataOps observability and the top DataOps tools businesses can use to streamline their data management, costs, and collaboration processes.

A wide variety of DataOps tools are available on the market, and choosing the right one can be a very daunting task. To help businesses make an informed decision, this article has compiled a list of the top DataOps tools that can be used to manage data-driven processes.

Data Integration Tools

1) Fivetran

Fivetran is a very popular and widely adopted data integration platform that simplifies the process of connecting various data sources to a centralized data warehouse [3]. This enables users or businesses to easily analyze and visualize their data in one place, eliminating the need to manually extract, transform, and load (ETL) data from multiple different sources.

Fivetran provides sets of pre-built connectors for a wide range of data sources, including popular databases, cloud applications, SaaS applications—and even flat files. These connectors automate the process of data extraction, ensuring that the data is always up-to-date, fresh and accurate. Once data is in the central data warehouse, Fivetran performs schema discovery and data validation, automatically creating tables and columns in the data warehouse based on the structure of the data source, making it really very easy to set up and maintain data pipelines without the need for manually writing custom code.

Fivetran also offers features like data deduplication, incremental data updates, and real-time data replication. These features help make sure that the data is always complete, fresh and accurate.

2) Talend Data Fabric

Talend Data Fabric solution is designed to help businesses and organizations make sure they have healthy data to stay in control, mitigate risk, and drive massive value. The platform combines data integration, integrity, and governance to deliver reliable data that businesses and organizations can rely on for decision-making processes. Talend helps businesses build customer loyalty, improve operational efficiency and modernize their IT infrastructure.

Talend's unique approach to data integration makes it easy for businesses and organizations to bring data together from multiple sources and power all their business decisions. It can integrate virtually any data type from any data source to any data destination(on-premises or in the cloud). The platform is flexible, allowing businesses and organizations to build data pipelines once and run them anywhere, with no vendor or platform lock-in. And the solution is an all-in-one (unified solution), bringing together data integration, data quality, and data sharing on an easy-to-use platform.

Talend's Data Fabric offers a multitude of best-in-class data integration capabilities, such as Data Integration, Pipeline Designer, Data Inventory, Data Preparation, Change Data Capture, and Data Stitching. These tools make data integration, data discovery/search and data sharing more manageable, enabling users to prepare and integrate data quickly, visualize it, keep it fresh, and move it securely.

3) StreamSets

StreamSets is a powerful data integration platform that allows businesses to control and manage data flow from a variety of batch and streaming sources to modern analytics platforms. You can deploy and scale your dataflows on-edge, on-premises, or in the cloud using its collaborative, visual pipeline design, while also mapping and monitoring them for end-to-end visibility[4]. The platform also allows for the enforcement of Data SLAs for high availability, quality, and privacy. StreamSets enables businesses and organizations to quickly launch projects by eliminating the need for specialized coding skills through its visual pipeline design, testing, and deployment features, all of which are accessible via an intuitive graphical user interface. With StreamSets, brittle pipelines and lost data will no longer be a concern, as the platform can automatically handle unexpected changes. The platform also includes a live map with metrics, alerting, and drill-down functionality, allowing businesses to efficiently integrate data in a breeze.

4) K2View

K2View provides enterprise-level DataOps tools. It offers a data fabric platform for real-time data integration, which enables businesses and organizations to deliver personalized experiences [6]. K2View's enterprise-level data integration tools integrate data from any kind of source and make it accessible to any consumer through various methods such as bulk ETL, reverse ETL, data streaming, data virtualization, log-based CDC, message-based integration, SQL—and APIs.

K2View can ingest data from various sources and systems, enhance it with real-time insights, convert it into its patented micro-database, and ensure performance, scalability, and security by compressing and encrypting the micro-database individually. It then applies data masking, transformation, enrichment, and orchestration tools on-the-fly to make the data accessible to authorized consumers in any format while adhering to data privacy and security rules.

5) Alteryx

Alteryx is a very powerful data integration platform that allows users to easily access, manipulate, analyze, and output data. The platform utilizes a drag-and-drop interface (low code/no code interface) and includes a variety of tools and connectors(80+) for data blending, predictive analytics, and data visualization[7]. It can be used in a one-off manner or, more commonly, as a recurring process called a "workflow." The way Alteryx builds workflows also serves as a form of process documentation, allowing users to view, collaborate, support and enhance the process. The platform can read and write data to files, databases, and APIs, and it also includes functionality for predictive analytics and geospatial analysis. Alteryx is currently being used in a variety of industries and functional areas and can be used to more quickly and efficiently automate data integration processes. Some common use cases include combining and manipulating data within spreadsheets, supplementing SQL development, APIs, cloud or hybrid access, data science, geospatial analysis—and creating reports and dashboards.

Note: Alteryx is often compared to ETL tools, but it is important to remember that its primary audience is data analysts. Alteryx aims to empower business users by giving them the freedom to access, manipulate, and analyze data without relying on IT.

Data Quality Testing and Monitoring Tools

1) Monte Carlo

Monte Carlo is a leading enterprise data monitoring and observability platform. It provides an end-to-end solution for monitoring and alerting for data issues across the data warehouses, data lakes, ETL, and business intelligence platforms. It uses machine learning and AI to learn about the data and proactively identify data-related issues, assess their impact, and notify those who need to know. The platform's automatic and immediate identification of the root cause of issues allows teams to collaborate and resolve problems faster, and it also provides automatic, field-level lineage , data discovery, and centralized data cataloging that allows teams to better understand the accessibility, location, health, and ownership of their data assets. The platform is designed with security in mind, scales accordingly with the provided stack, and includes a no-code or low-code(code-free) onboarding feature for easy implementation with the existing data stack.

2) Databand

Databand is a data monitoring and observability platform recently acquired by IBM that helps organizations detect and resolve data issues before they impact the business. It provides a fierce, end-to-end view of data pipelines, starting with source data, which allows businesses and organizations to detect and resolve issues early, reducing the mean time to detection (MTTD) and mean time to resolution (MTTR) from days and weeks to minutes.

One key features of Databand is its ability to automatically collect metadata from modern data stacks such as Airflow, Spark, Databricks, Redshift, dbt, and Snowflake. This metadata is used to build historical baselines of common data pipeline behavior, which allows organizations to get visibility into every data flow from source to destination.

Databand also provides incident management, end-to-end lineage, data reliability monitoring, data quality metrics, anomaly detection, and DataOps alerting and routing capabilities. With this, businesses and organizations can improve data reliability and quality and visualize how data incidents impact upstream and downstream components of the data stack. Databand's combined capabilities provide a single solution for all data incidents, allowing engineers to focus on building their modern data stack rather than fixing it.

3) Data Fold

Datafold is a data reliability platform focused on proactive data quality management that helps businesses prevent data catastrophes. It has the unique ability to detect, evaluate, and investigate data quality problems before they impact productivity. The platform offers real-time monitoring to identify issues quickly and prevent them from becoming data catastrophes.

Datafold harnesses the power of machine learning with AI to provide analytics with real-time insights, allowing data engineers to make top-quality predictions from large amounts of data.

Some of the key features of Datafold include:

• One-Click Regression Testing for ETL

• Data flow visibility Across all pipelines and BI reports

• SQL Query Conversion, Data Discovery, and Multiple data source Integrations

Datafold offers a simple yet intuitive user interface(UI) and navigation with powerful features. The platform allows deep exploration of how tables and data assets relate. The visualizations are really very easy to understand. Data quality monitoring is also super flexible. However, the data integrations they support are relatively limited.

4) Query Surge

QuerySurge is a very powerful/versatile tool for automating data quality testing and monitoring, particularly for big data, data warehouses, BI reports, and enterprise-level applications. It is particularly designed to integrate seamlessly, allowing for continuous testing and validation of data as it flows.

Query Surge also provides the ability to create and run tests without needing to write SQL through smart query wizards. This allows for column, table, and row-level comparisons and automatic column matching. Also, users can create custom tests that can be modularized with reusable "snippets" of code, set thresholds, check data types, and perform other advanced number of validation checks. QuerySurge also has robust scheduling capabilities, allowing users to run tests pronto, at a specified date and time. On top of that, it also supports 200+ supported vendors and tech stacks, so it can test across a wide variety of platforms, including big data lakes, data warehouses, traditional databases, NoSQL document stores, BI reports, flat files, JSON files—and a whole lot more.

One key benefits of QuerySurge is its ability to integrate with other solutions in the DataOps pipeline, such as data integration/ETL solutions, build/configuration solutions, QA and test management solutions. The tool also includes a Data Analytics Dashboard, which allows users to monitor test execution progress in real-time, drill down into data to examine results, and see stats for executed tests. It also has an out-of-the-box integration with plethora of services and any other solution with API access.

QuerySurge is available both on-premises and in the cloud, with support for AES 256-bit encryption, LDAP/LDAPS, TLS, HTTPS/SSL, auto-timeout, and other security features. In a nutshell, QuerySurge is a very powerful and comprehensive solution for automating data monitoring and testing, allowing businesses and organizations to improve their data quality at speed and reduce the risk of data-related issues in the delivery pipeline.

5) Right Data

Right Data's RDT is a powerful data testing and monitoring platform that helps businesses and organizations improve the reliability and trust of their data by providing an easy-to-use interface for data testing, reconciliation, and validation. It allows users to quickly identify issues related to data consistency, quality, and completeness. It also provides an efficient way to analyze, design, build, execute and automate reconciliation and validation scenarios with little to no coding required, which helps save time and resources.

Key features of RDT:

• Ability to analyze DB: It provides a full set of applications to analyze the source and target datasets. Its top-of-the-line Query Builder and Data Profiling features help users understand and analyze the data before they use the corresponding datasets in different scenarios.

• Support of a wide range of data sources: RDT supports a wide range of data sources such as ODBC or JDBC, flat files, cloud technologies, SAP, big data, BI reporting—and various other sources. This allows businesses and organizations to easily connect to and work with their existing data source.

• Data reconciliation: RDT has features like "Compare Row Counts" that let users compare the number of rows in the source dataset and the target dataset and find tables where the number of rows doesn't match. It also provides a "row-level data compare" feature that compares datasets between source/target and identifies rows that do not match each other.

• Data Validation: RDT provides a user-friendly interface for creating validation scenarios, which enables users to establish one or more validation rules for target data sets, identify exceptions, and analyze and report on the results.

• Admin & CMS: RDT has an admin console that allows the admin to manage and config the features of the tool. The console provides the ability to create + manage users, roles, and the mapping of roles to specific users. Administrators can also create, manage, and test connection profiles, which are used to create queries. The tool also provides a Content Management Studio (CMS) that enables exporting of queries, scenarios, and connection profiles from one RightData instance to another. This feature is useful for copying within the same instance from one folder to another and also for switching over the connection profile of queries.

DataOps Observability and Augmented FinOps

1) Chaos Genius

Chaos Genius is a powerful DataOps Observability tool that uses ML and AI to provide precise cost projections and enhanced metrics for monitoring and analyzing data and business metrics. One of the main reasons the tool was built was to provide value to businesses by offering a powerful, first-in-class DataOps observability tool that can help monitor and analyze data, lower spending, and improve business metrics. The tool utilizes machine learning and artificial intelligence (ML/AI) to sift through data and provide more precise cost projections and enhanced metrics.

Chaos Genius currently offers a service called "Snowflake Observability" as one of its main services.

Key features of Chaos Genius (Snowflake Observability) include:

• Cost optimization and monitoring: Chaos Genius is designed to help businesses and organizations optimize and monitor the cost of a Snowflake cloud data platform. This includes finding places where costs can be cut and making suggestions for how to do so.

• Enhanced query performance: Chaos Genius can analyze query patterns to identify inefficient queries and make smart recommendations to improve their performance, which can lead to faster and more efficient data retrieval and improve the overall performance of the data warehouse.

• Reduced Spendings: Chaos Genius enables businesses to better enhance the efficiency of their systems and reduce total spending by about ~10% - 30%.

• Affordability: Chaos Genius offers an affordable pricing model with three tiers. The first tier is completely free, while the other two are business-oriented plans for companies that want to monitor more metrics. This makes it accessible to businesses of all sizes and budgets.

2) Unravel

Unravel is a DataOps observability platform that provides businesses and organizations with a thorough view of their entire data stack and helps them optimize performance, automate troubleshooting, and manage and monitor the cost of their entire data pipelines. The platform is also designed to work with different cloud service providers, for example, Azure, Amazon EMR, GCP, Cloudera and even on-premises environments, providing businesses with the flexibility to manage their data pipeline regardless of where their data resides.

Unravel uses the power of machine learning and AI to model data pipelines from end to end, providing businesses with a detailed understanding of how data flows through their systems. This enables businesses/organizations to identify bottlenecks, optimize resource allocation and improve the overall performance of their data pipelines.

The platform's data model enables businesses to explore, correlate, and analyze data across their entire environment, providing deep insights into how apps, services, and resources are used and what works and what doesn't, allowing businesses to quickly identify potential issues and take immediate action to resolve them. Not only that, but Unravel also has automatic troubleshooting features that can help businesses find the cause of a problem quickly and take steps to fix it, saving them a huge amount of spending and making their data pipelines more reliable and efficient.

Data Orchestration Tools

1) Apache Airflow

Apache Airflow is a fully open source DataOps workflow orchestration tool to author, schedule, and monitor workflows programmatically. Airbnb first developed it, but now it is under the Apache Software Foundation [8]. It is a tool for expressing and managing data pipelines and is often used in data engineering. It allows users to define, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Airflow provides a simple and powerful way to manage data pipelines, and it is simple to use, allowing users to create and manage complex workflows quickly; on top of that, it has a large and active community that provides many plugins, connectors, and integrations with other tools that makes it very versatile.

Key features of Airflow include:

• Dynamic pipeline generation: Airflow's dynamic pipeline generation is one of its key features. Airflow allows you to define and generate pipelines programmatically rather than manually creating and managing them. This facilitates the creation and modification of complex workflows.

• Extensibility: Airflow allows using custom plugins, operators and executors, which means you can add new functionality to the platform to suit your specific needs and requirements; this makes Airflow highly extensible and an excellent choice for businesses and organizations with unique requirements or working with complex data pipelines.

• Scalability: Airflow has built-in support for scaling thousands of tasks, making it very well-suited for large-scale organizations or running large-scale data processing tasks.

2) Shipyard

Shipyard is a powerful data orchestration tool designed to help data teams streamline and simplify their workflows and deliver data at very high speed. The tool is intended to be code-agnostic, allowing teams to deploy code in any language they prefer without the need for a steep learning curve. It is cloud-ready, meaning it eliminates the need for teams to spend hours and hours spinning up and managing their servers. Instead, they can orchestrate their workflows in the cloud, allowing them to focus on what they do best—working with data. Shipyard can also run thousands of jobs at once, making it ideal for scaling data processing tasks. The tool can dynamically scale to meet the demand, ensuring that workflows run smoothly and efficiently even when dealing with large amounts of data.

Shipyard comes with a very intuitive visual UI, allowing users to construct workflows directly from the interface and make changes as needed by dragging and dropping. The advanced scheduling, webhooks and on-demand triggers make automating workflows on any schedule easy. It also allows for cross-functional workflows, meaning that the entire data process can be interconnected across the entire data lifecycle, helping teams keep track of the entire data journey, from data collection and processing to visualization and analysis.

Shipyard also provides instant notifications, which help teams catch and fix critical breakages before anyone even notices. It also has automatic retries and cutoffs, which give workflows resilience, so teams don't have to lift a finger. Not only that, it can isolate and address the root cause in real time, so teams can get back up and running in seconds. Also, it allows teams to connect their entire data stack in minutes, seamlessly moving data between the existing tools in the data stack, regardless of the cloud provider. With over 20+ integrations and 60+ low-code templates to choose from, data teams can begin connecting their existing tools in record speeed!!!

3) Dagster

Dagster is a next-generation open source data orchestration platform for developing, producing, and observing data assets in real-time. Its primary focus is to provide a unified experience for data engineers, data scientists, and developers to manage the entire lifecycle of data assets, from development and testing to production and monitoring. Using Dagster, users can manage their data assets with code and monitor "runs" across all jobs in one place with the run timeline view. On the other hand, the run details view allows users to zoom into a run and pin down issues with surgical precision.

Dagster also allows users to see each asset's context and update it all in one place, including materializations, lineage, schema, schedule, partitions—and a whole lot more. Not only that, but it also allows users to launch and monitor backfills over every partition of data. Dagster is an enterprise-level orchestration platform that prioritizes developer experience(DX) with fully serverless + hybrid deployments, native branching, and provides out-of-the-box CI/CD configuration.

4) AWS glue

AWS Glue is a data orchestration tool that makes it easy to discover, prepare, and combine data for analytics and machine learning workflows. With Glue, you can crawl data sources, extract, transform and load (ETL) data, and create/schedule data pipelines using a simple visual UI interface. Glue can also be used for analytics and includes tools for authoring, running jobs, and implementing business workflows. AWS Glue offers data discovery, ETL, cleansing, and central cataloging and allows you to connect to over 70 diverse data sources [9]. You can create, run and monitor ETL pipelines to load data into data lakes and query cataloged data using Amazon Athena, Amazon EMR, and Redshift Spectrum. It is serverless in nature, meaning there's no infrastructure to manage, and it supports all kinds of workloads like ETL, ELT, and streaming all packaged in one service. AWS Glue is very user-friendly and is suitable for all kinds of users, including developers and business users. Its ability to scale on demand allows users to focus on high-value activities that extract maximum value from their data; it can handle any data size and support all types of data and schema variations.

AWS Glue provides TONS of awesome features that can be used in a DataOps workflow, such as:

• Data Catalog: A central repository to store structural and operational metadata for all data assets.

• ETL Jobs: The ability to define, schedule, and run ETL jobs to prepare data for analytics.

• Data Crawlers: Automated data discovery and classification that can connect to data sources, extract metadata, and create table definitions in the Data Catalog.

• Data Classifiers: The ability to recognize and classify specific types of data, such as JSON, CSV, and Parquet.

• Data Wrangler: A visual data transformation tool that makes it easy to clean and prepare data for analytics.

• Security: Integrations with AWS Identity and Access Management (IAM) and Amazon Virtual Private Cloud (VPC) to help secure data in transit and at rest.

• Scalability: The ability to handle petabyte-scale data and thousands of concurrent ETL jobs.

Data Governance Tools

1) Collibra

Collibra is an enterprise-oriented data governance tool that helps businesses and organizations understand and manage their data assets. It enables businesses and organizations to create an inventory of data assets, capture metadata about 'em, and govern these assets to ensure regulatory compliance. The tool is primarily used by IT, data owners, and administrators who are in charge of data protection and compliance to inventory and track how data is used. Collibra's main aim is to protect data, ensure it is appropriately governed and used, and eliminate potential fines and risks from a lack of regulatory compliance.

Collibra offers six key functional areas to aid in data governance:

• Collibra Data Quality & Observability: Monitors data quality and pipeline reliability to aid in remedying anomalies.

• Collibra Data Catalog: A single solution for finding and understanding data from various sources.

• Data Governance: A location for finding, understanding, and creating a shared language around data for all individuals within an organization.

• Data Lineage: Automatically maps relationships between systems, applications, and reports to provide a comprehensive view of data across the enterprise.

• Collibra Protect: Allows for the discovery, definition, and protection of data from a unified platform.

• Data Privacy: Centralizes, automates, and guides workflows to encourage collaboration and address global regulatory requirements for data privacy.

2) Alation

Alation is an enterprise-level data catalog tool that serves as a single reference point for all of an organization's data. It automatically crawls and indexes over 60 different data sources, including on-premises databases, cloud storage, file systems, and BI tools. Using query log ingestion, Alation parses queries to identify the most frequently used data and the individuals who use it the most, forming the basis of the catalog. Users can then collaborate and provide context for the data. With the catalog in place, data analysts and scientists can quickly and easily locate, examine, verify, and reuse data, hence boosting their productivity. Alation can also be used for data governance, allowing analytics teams to efficiently manage and enforce policies for data consumers.

Key benefits of using Alation:

• Boost analyst productivity

• Improve data comprehension

• Foster collaboration

• Minimize the risk of data misuse

• Eliminate IT bottlenecks

• Easily expose and interpret data policies

Alation offers various solutions to improve productivity, accuracy and data-driven decision-making. These include:

• Alation Data Catalog: Improves the efficiency of analysts and the accuracy of analytics, empowering all members of an organization to find, understand, and govern data efficiently.

• Alation Connectors: A wide range of native data sources that speed up the process of gaining insights and enable data intelligence throughout the enterprise. (Additional data sources can also be connected with the Open Connector Framework SDK.)

• Alation Platform: An open and intelligent solution for various metadata management applications, including search and discovery, data governance, and digital transformation.

• Alation Data Governance App: Simplifies secure access to the best data in hybrid and multi-cloud environments.

• Alation Cloud Service: Offers businesses and organizations the option to manage their data catalog on their own or have it managed for them in the cloud.

Data Cloud and Data Lake Platforms

1). Databricks

Databricks is a cloud-based lakehouse platform founded in 2013 by the creators of Apache Spark, Delta Lake, and MlFlow [10]. It unifies data warehousing and data lakes to provide an open and unified platform for data and AI. The Databricks Lakehouse architecture is designed to manage all data types and is cloud-agnostic, allowing data to be governed wherever it is stored. Teams can collaborate and access all the data they need to innovate and improve. The platform includes the reliability and performance of Delta Lake as the data lake foundation, fine-grained governance and support for persona-based use cases. It also provides instant and serverless compute, managed by Databricks. The Lakehouse platform eliminates the challenges caused by traditional data environments such as data silos and complicated data structures. It is simple, open, multi-cloud, and supports various data team workloads. The platform allows for flexibility with existing infrastructure, open source projects, and the Databricks partner network.

2) Snowflake

Snowflake is a cloud data platform offering a software-as-a-service model fo storing and analyzing LARGE amounts of data. It is designed to support high levels of concurrency, scalability and performance. It allows customers to focus on getting value from their data rather than managing the infrastructure on which it's stored. The company was founded in 2012 by three experts, Benoit Dashville, Thierry Cruanes, and Marcin Zukowski [11]. Snowflake runs on top of cloud infrastructure, such as AWS, Microsoft Azure, and Google's cloud platforms. It allows customers to store and analyze their data using the elasticity of the cloud, providing speed, ease of use, cost-effectiveness, and scalability. It is widely used for data warehousing, data lakes, and data engineering. It is designed to handle the complexities of modern data management processes. Not only that, but it also supports various data analytics applications, such as BI tools, ML/AI, and data science. Snowflake also revolutionized the pricing model by utilizing a "utilization model" that focuses on a client's consumption based on whether they're computing or storing data, making everything more flexible and elastic.

Key features of Snowflake include:

• Scalability: Snowflake offers scalability through its multi-cluster shared data architecture, allowing for easy scaling up and down of resources as needed.

• Cloud-Agnostic: Snowflake is available on all major cloud providers (AWS, GCP, AZURE) while maintaining the same user experience, allowing for easy integration with current cloud architecture.

• Auto-scaling + Auto-Suspend: Snowflake automatically starts and stops clusters during resource-intensive processing and stops virtual warehouses when idle for cost and performance optimization.

• Concurrency and Workload Separation: Snowflake's multi-cluster architecture separates workloads to eliminate concurrency issues and ensures that queries from one virtual warehouse will not affect another.

• Zero Hardware + Software config: Snowflake does not require software installation or hardware config or commissioning, making it easy to set up and manage.

• Semi-Structured Data: Snowflake's architecture allows for the storage of structured and semi-structured data through the use of VARIANT data types.

• Security: Snowflake offers a wide range of security features, including network policies, authentication methods and access controls, to ensure secure data access and storage.

4) Google Bigquery

Google BigQuery is a fully-managed and serverless data warehouse provided by Google Cloud that helps organizations manage and analyze large amounts of data with built-in features such as machine learning, geospatial analysis, and business intelligence[12]. It allows businesses and organizations to easily store, ingest, store, analyze, and visualize large amounts of data. Bigquery is designed to handle up to petabyte-scale data and supports SQL queries for data analysis purposes. The platform also includes BigQuery ML, which allows businesses or users to train and execute machine learning models using their enterprise data without needing to move it around.

BigQuery integrates with various business intelligence tools and can be easily accessed through the cloud console, a command-line tool, and even APIs. It is also directly integrated with Google Cloud’s Identity and Access Management Service so that one can securely share data and analytics insights across organizations. With BigQuery, businesses only have to pay for data storing, querying, and streaming inserts. Loading and exporting data are absolutely free of charge.

3) Amazon Redshift

Amazon Redshift is a cloud-based data warehouse service that allows for the storage and analysis of large data sets. It is also useful for migrating LARGE databases. The service is fully managed and offers scalability and cost-effectiveness for storing and analyzing large amounts of data. It utilizes SQL to analyze structured and semi-structured data from a variety of sources, including data warehouses, operational databases, and data lakes, which are enabled by AWS-designed hardware and powered by AI & machine learning; due to this, it is able to deliver optimal cost-performance at any scale. The service also offers high-speed performance and efficient querying capabilities to assist in making business decisions.

Key features of Amazon Redshift include:

• High Scalability: Redshift allows users to start with a very small amount of data and scale up to a petabyte or more as their data grows incrementally.

• Query execution + Performance: Redshift uses columnar storage, advanced compression, and parallel query execution to deliver fast query performance on large data sets.

• Pay-as-you-go pricing model: Redshift uses a pay-as-you-go pricing model and allows users to choose from a range of node types and sizes to optimize cost and performance.

• Robust Security: Redshift integrates with AWS security services like AWS Identity and Access Management (IAM) and Amazon Virtual Private Cloud (VPC)—and more(learn more from here)—to keep data safe.

• Integration: Redshift can be easily integrated with varios other services such as Datacoral, Etleap, Fivetran, SnapLogic,Stitch,Upsolver,Matillion—and more.

• Monitoring + Management tools: Redshift has various management and monitoring tools, including the Redshift Management Console and Redshift Query Performance Insights, to help users manage and monitor their clusters in their data warehouse.

Conclusion

As the amount of data continues to grow at an unprecedented rate, the need for efficient data management and observability solutions has never been greater. But simply collecting and storing data won't cut it—it's the insights and value it can provide that truly matter. However, this can only be achieved if the data is high quality, up-to-date, and easily accessible. This is exactly where DataOps comes in—providing a powerful set of best practices and tools to improve collaboration, integration, and automation, allowing businesses to streamline their data pipelines, reduce costs and workload, and enhance data quality. Hence, by utilizing the tools mentioned above, businesses can minimize data-related expenses and extract maximum value from their data.

Don't let your data go to waste—harness its power with DataOps.

References

[1]. A. Dyck, R. Penners and H. Lichter, "Towards Definitions for Release Engineering and DevOps," 2015 IEEE/ACM 3rd International Workshop on Release Engineering, Florence, Italy, 2015, pp. 3-3, doi: 10.1109/RELENG.2015.10.

[2] Doyle, Kerry. “DataOps vs. MLOps: Streamline your data operations.” TechTarget, 15 February 2022, https://www.techtarget.com/searchitoperations/tip/DataOps-vs-MLOps-Streamline-your-data-operations. Accessed 12 January 2023.

[3] Danise, Amy, and Bruce Rogers. “Fivetran Innovates Data Integration Tools Market.” Forbes, 11 January 2022, https://www.forbes.com/sites/brucerogers/2022/01/11/fivetran-innovates-data-integration-tools-market/. Accessed 13 January 2023.

[4] Basu, Kirit. “What Is StreamSets? Data Engineering for DataOps.” StreamSets, 5 October 2015, https://streamsets.com/blog/what-is-streamsets/. Accessed 13 January 2023.

[5] Chand, Swatee. “What is Talend | Introduction to Talend ETL Tool.” Edureka, 29 November 2021, https://www.edureka.co/blog/what-is-talend-tool/#WhatIsTalend. Accessed 12 January 2023.

[6] “Delivering real-time data products to accelerate digital business [white paper].” K2View, https://www.k2view.com/hubfs/K2View%20Overview%202022.pdf. Accessed 13 January 2023.

[7] “Complete introduction to Alteryx.” GeeksforGeeks, 3 June 2022, https://www.geeksforgeeks.org/complete-introduction-to-alteryx/. Accessed 13 January 2023.

[8] “Apache Airflow: Use Cases, Architecture, and Best Practices.” Run:AI, https://www.run.ai/guides/machine-learning-operations/apache-airflow. Accessed 12 January 2023.

[9] “What is AWS Glue? - AWS Glue.” AWS Documentation, https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html. Accessed 13 January 2023.

[10] “About Databricks, founded by the original creators of Apache Spark™.” Databricks, https://www.databricks.com/company/about-us. Accessed 18 January 2023.

[11] “You're never too old to excel: How Snowflake thrives with 'dinosaur' cofounders and a 60-year-old CEO.” LinkedIn, 4 September 2019, https://www.linkedin.com/pulse/youre-never-too-old-excel-how-snowflake-thrives-dinosaur-anders/. Accessed 18 January 2023.

[12] “What is BigQuery?” Google Cloud, https://cloud.google.com/bigquery/docs/introduction. Accessed 18 January 2023.

In this article, we'll provide a comprehensive introduction and guide to DataOps, covering its essential key components, the benefits it offers, how it differs from DevOps and the best practices for implementing it. We'll also go over some of the potential challenges in implementing DataOps and provide resources for further reading on this vital data management operations strategy.

But first, let's define DataOps and explain why it's become such a crucial part of modern data management.

What is DataOps?

Data Operations, or DataOps for short, is used to describe a set of practices and processes that are designed to improve the collaboration, integration, and automation of data management operations and tasks [1]. These practices and processes include a focus on agile methodologies. It is intended to help organizations better manage their data pipelines, reduce the workload and time required to develop and deploy new data-driven applications and improve the quality of the data being used. DataOps is meant to eliminate barriers between data engineers, data scientists and data/business analysts—as well as other teams and departments within an organization—enabling them to work together more efficiently and effectively to manage and analyze data.

Many businesses and organizations have already adopted DataOps principles to make better use of their data and increase productivity [2]. Let's take a look at "Netflix" as an example; they have a large and very complex data environment, with data coming from multiple different sources, including subscriber accounts, viewing or streaming activity, and customer support inquiries. To manage this data effectively, Netflix has implemented DataOps practices and tools, such as automation, collaboration, and monitoring. Netflix has automated the data ingestion, automation and preparation processes, allowing it to quickly and accurately integrate data from multiple different sources and prepare it for analysis, which will help Netflix directly gain a better understanding of subscriber activity, behaviour and preferences, which in turn allows it to make better decisions about content/movie/show recommendations, their own marketing campaigns, and product development.

Why is DataOps important?

In today's fast-paced modern business world, DataOps plays a vital role in helping businesses and organizations stay ahead, as the ability to analyze data rapidly and precisely can provide them with a competitive advantage over others. DataOps simplifies and automates the complex process of collecting, storing, and analyzing data, making it more efficient, accurate, and relevant to the business's needs/requirements. This enables businesses to make better use of their data assets and derive more value from them. Overall, DataOps plays a key part in any organization's data management and data management operations strategy because it lets them use their fresh data assets to drive business growth and fresh new innovations.

DataOps empowers businesses and organizations to make better, faster decisions and get the most out of their data. It helps them extract valuable insights—and drive productivity as a result. With the right tools and a well-thought-out plan, (businesses) can make more informed timely decisions.[3] Nevertheless, a significant obstacle to data-driven initiatives is ensuring decision-makers have access to important data and know how to use it effectively[4]. DataOps helps bridge this gap and fosters collaboration between teams, thereby enabling organizations to deliver products faster and more effectively.

DataOps is a people-driven practice, meaning that it depends on the abilities and knowledge of the individuals. It is not a tool or application that can be bought and implemented without the required human resources. Instead, it necessitates a team of proficient data experts that can collaborate effectively and efficiently [7].

Exploring the Key Differences Between DevOps and DataOps

DevOps is a set of practices that combines software development (Dev) and IT operations (Ops) to reduce system and application development lifecycles. DevOps has been defined as an organizational approach aimed at creating empathy and cross-functional collaboration [5]. It aims to establish an environment in which software development, building, testing, and release can occur more quickly, frequently, and consistently.

The main goal of DevOps is to improve the collaboration and communication between developers and operations teams and automate the build, test, and release service cycle and manage and monitor infrastructure and applications in production.

DevOps Lifecycle

DevOps lifecycle consists of several phases that are followed when developing and maintaining software applications.

• Plan: The plan phase involves identifying the goals/objectives of the project and the resources that will be required to complete it.

• Develop: Develop phase where the software is actually developed. This involves writing code, building mockups/prototypes, and testing the software to ensure it is functional.

• Test: The test phase comes after the software has been developed; it must be tested to ensure it is error-free and will function as intended. This may include unit testing, integration testing—and other types of testing.

• Deploy: Deploy phase is where the software or application is deployed to a production environment where end users can use it.

• Maintain: Maintain phase is where the software will be maintained to guarantee it continues to work as expected. This could involve patch fixes, security upgrades and hotfixes to ensure the software runs smoothly indefinitely.

DataOps Lifecycle

The DataOps lifecycle typically consists of the following stages:

• Ingest: Ingest stage involves extracting data from multiple different raw data sources and storing it in a centralized location, such as a data warehouse or data lake.

• Prepare: Prepare stage is where data engineers and data scientists prepare the data for analysis by extracting, cleaning, and transforming it. This may involve tasks such as data deduplication, data integration/mining and feature extraction.

• Model: The model **stage involves building AI/ML models and other statistical models to analyze and make predictions based on the data. Data scientists are typically responsible for this stage.

• Visualize: Visualize stage involves creating charts, graphs and other visualizations to help others understand and interpret the data.

• Deploy: Deploy stage is where the models and other data products developed in previous stages are deployed and made available to end users.

• Observability: The observability stage involves monitoring and analyzing the performance of the data quality and ensuring that it meets the needs of the end users. This stage also involves collecting feedback and implementing improvements as needed.

To sum up, now that you know what sets the DataOps and DevOps lifecycles apart, DataOps seeks to optimize an organization's whole data lifecycle, from data ingestion and preparation through analysis and visualization. In contrast, DevOps is focused on enhancing the agility of the software development process through automation and integration. DataOps aims to improve the efficiency and effectiveness of data processing and utilization. It can be thought of as the function within an organization that controls the data journey from source to value[6].

Collaboration Across Teams for Data Delivery

DataOps is a collaborative effort within an organization, with many different teams of people working together to ensure that DataOps functions properly and delivers data value [3]. So, before the data is delivered to end users, it is subjected to a number of treatments and refinements from multiple teams. Data scientists first use their data science techniques, such as machine learning and deep learning to build models using software stacks such as Python or R and tools such as Spark or Tensorflow, among others, and the models are then transferred to data engineers, who collect and manage the data used to train and evaluate these models, while data developers and data architects create complete applications that include the models. The data governance team then implements data access controls for training and benchmarking purposes, while the operations team ( "Ops") is in charge of putting everything together and making it available to end users.

Key Components of DataOps

DataOps involves several key components which work together to improve data management processes. These includes:

Continuous integration and Continuous delivery (CI/CD)

Continuous integration and Continuous delivery (CI/CD) is a practice that involves frequently integrating and testing code changes and then quickly and efficiently pushing those changes to production environments. In DataOps, CI/CD plays a crucial role in ensuring the accuracy and consistency of data as it is integrated and delivered to the appropriate people/systems. By constantly developing, building, automating and testing data changes and then quickly delivering them to production without any downtime, DataOps teams can minimize the risk of errors and ensure that data is delivered in a timely and reliable manner.

Data governance

The process of establishing policies, procedures, and standards for managing data assets, as well as an organizational structure to support enterprise data management, is known as data governance. Data governance in DataOps helps to ensure that data is collected, stored and used in a consistent and in ethical manner.

Data quality management and measurement

Data quality management and measurement involve identifying, correcting, and preventing any errors or inconsistencies in data. It helps ensure that the data being used is fully reliable—and accurate. This is critical because poor data quality can lead to incorrect or misleading insights and decisions, which can have serious consequences.

Data Orchestration

Data orchestration refers to the management and coordination of data processing tasks in a data pipeline. It involves specifying and scheduling how tasks will be completed, as well as dealing with errors and how tasks interact with one another. Data orchestration is critical in DataOps for automating and optimizing the flow of data through the pipeline. This can include tasks such as extracting data from various sources, transforming and cleaning the data, and loading it into a target system for analysis or reporting purposes.

DataOps Observability

As we’ve already discussed what DataOps is, let’s briefly review it. DataOps is a collection of best practices and technology used to manage and develop data products, optimize data management processes, improve quality, speed, and collaboration and promote continuous improvement. DataOps is based on the same principles and practices as DevOps. Still, it has taken longer to become fully matured because data is constantly changing and can be more fragile than software applications/infrastructures. For example, let's suppose that if a software application goes down, it can be easily restored without significant impact, but if data becomes corrupted, it may have serious consequences. This is the exact reason why DataOps has taken longer to get off the ground compared to DevOps.

To ensure that data performs optimally and meets desired standards for quality, reliability, and efficiency, it is important to implement DataOps observability. This involves regularly observing and monitoring data and using the insights gained to make informed decisions. DataOps observability is a newer concept, but the practice of observability itself has a long history in the DevOps world. For example, observability platforms/solutions such as AppDynamics and Splunk help software engineers improve application reliability and reduce site/app downtime.

DataOps observability is not just limited to testing and monitoring data quality and the data pipeline. It also includes monitoring the health of the machine learning models, analyzing the application security measures to data infrastructure, tracking KPI and business monitoring. In other words, it covers a wide range of areas beyond just monitoring the health of data quality and data pipeline.

DataOps observability is a somewhat ambiguous concept that is interpreted differently in the data community. Still, in essence, it refers to an organization's/businesses’ ability to fully understand the health of the data. To sum it up, DataOps observability must address a few key areas: data quality and data pipeline reliability. Data quality is important to business users who want high-quality data they can trust. Data pipeline reliability is critical to data engineers and scientists, who need their data pipeline to run smoothly. Also, In addition to these two components, DataOps observability includes BizOps, which tracks/monitors the health and KPI of the business, as well as monitors the usage and the cost of the data. This is exactly where Chaos Genius fits in. Offering a complete observability solution, it facilitates businesses and organizations in testing the resilience and reliability of data, which can directly help businesses to improve their spending and boost their performance.

To create a successful data product, businesses should focus on three key areas: data governance, data access and security, and DataOps and quality.

Data governance involves understanding where the data comes from, while data access and security ensure that the data is being used in an appropriate and secure manner. Finally, DataOps and Quality involve automation, orchestration, CI/CD, configuration management and observability to ensure that the data product is high quality. The unification of these use cases is essential for the success of the data product.

In a nutshell, "DataOps Observability" refers to the ability to monitor and understand the various processes and systems involved in data management, with the main goal of ensuring the reliability, trustworthiness, and business value of the data. It involves monitoring and analyzing data pipelines, ensuring the quality of the data and demonstrating the business value of the data through metrics like financial and operational efficiency. DataOps observability allows businesses to improve the efficiency of their data management processes and make better use of their data assets. It helps to ensure that data is accurate, reliable, and easily accessible, enabling businesses and organizations to make data-driven decisions and drive business value.

Implementing DataOps

Implementing DataOps involves following a number of steps to ensure that data is collected, stored, and used in a way that supports business goals/objectives. This starts by identifying the data requirements and specifying the sources and types of data needed. A data governance framework is then established to ensure that data is collected, stored and used in a consistent and compliant manner. Data pipelines are designed and implemented to extract, transform, and load data from various sources into a centralized repository, and data quality checks and monitoring are put in place to ensure the accuracy, completeness and consistency of the data. To support a data-driven culture, it is crucial to build a collaborative and cross-functional team and establish a focus on data literacy, continuous improvement, and data-driven decision-making. Finally, it is important to continuously monitor and optimize the DataOps processes to improve efficiency, effectiveness, and agility.

List of Top DataOps tools and platforms available

One of the key components of DataOps is the use of specialized tools to manage and automate the flow of data. Tools can help with tasks such as scheduling and monitoring the execution of data pipelines, extracting, transforming and cleaning data, and integrating data from multiple different sources. There are various different DataOps tools available on the market, and the best choice will depend on your specific needs/requirements. Some tools are designed for general-purpose data integration and transformation, while others are more specialized for specific types of data or use cases. Here are some of the TOP Trending and Popular DataOps tools currently available on the market.

Apache Airflow:

Apache airflow is an open-source tool that is used for scheduling, monitoring and managing the execution of data pipelines. It provides a simple, intuitive interface for defining and organizing tasks, and it can be extended with custom plugins to support a wide range of data sources and destinations.

Databricks

Databricks is a cloud-based platform for data engineering, data science and AI/ML. It is built on top of the Apache Spark big data processing engine and offers a variety of tools for working with large amounts of data. Databricks' collaborative workspace is a great way for groups to collaborate on data projects together in real time. It provides a fully web-based notebook-like environment for writing and executing code, as well as data exploration and visualization tools. Databricks consist of connectors for common data sources and destinations, a library of pre-built transformations and functions, and support for different programming languages.

Snowflake

Snowflake is not a DataOps tool per se, it's a platform that can be used as a foundation for DataOps. Snowflake is a cloud-based data storage and analytics platform that is widely used for data warehousing, data lakes and data engineering. It is designed to handle the complexities of modern data management processes, such as data integration, data quality, data security, and data governance and to support a variety of data analytics applications, such as BI tools, ML and data science. Snowflake can also be used to manage the flow of data from various sources to the data warehouse, where it can be transformed, cleansed and optimized accordingly for analysis purposes. Snowflake’s architecture is designed to support high levels of concurrency, scalability and performance, making it well-suited for handling large amounts of data in real time. It also provides a number of features that supports data governance and security, such as data lineage, masking and auditing, which can be a very important consideration in DataOps environments.

Fivetran

Fivetran is also a cloud-based service that simplifies the process of transferring data between various sources and destinations(including Snowflake). It includes a range of connectors for popular data sources and destinations, including databases, cloud storage, SaaS applications—and more. One of the main features of Fivetran is its ability to support real-time synchronization and incremental updates, which means that it can continuously transfer new and updated data. This makes it ideal for use in scenarios where data needs to be kept up-to-date in near real-time. Fivetran has the ability to transfer data, but it also has a number of tools for managing and monitoring data pipelines. These tools include a web-based dashboard for tracking the status of data transfers and alerts for detecting issues and fixing 'em.

Talend

Talend is a commercial data integration platform that offers a wide range of tools for extracting, transforming, and loading (ETL) data. It includes an awesome and highly interactive graphical user interface (GUI) for building data pipelines and a library of pre-built connectors and transformations that can be used to integrate data from a wide range of sources and destinations. One of the main key features of Talend is its support for a wide range of data sources and destinations, including databases, cloud storage, SaaS applications—and more. It also includes support for popular programming languages, which allows users to write custom transformations and integrations. Talend also provides a range of tools for data governance, data quality, and data management, including support for tracking and managing data lineage.

Learn more about the Top DataOps tools available on the market in 2023.

Future of DataOps

DataOps is constantly evolving in response to emerging technologies and changing business needs. According to a report by MarketBiz, the global DataOps platform market is expected to experience significant growth over the forecast period of 2023 - 2032, with a projected value of $7,091.38 million. This growth is driven by the increasing demand for real-time data insights, the adoption of cloud-based solutions and the rising popularity of Agile and DevOps-related practices. The DataOps platform market is also anticipated to see growth in various regions, including North America, Europe, Asia Pacific, Latin America, the Middle East—and Africa. The market is projected to reach a value of $7,091.38 million, up from $1,150 million in 2022, with a compound annual growth rate of 22.4%.

The future of DataOps looks VERY bright with the current adoption of automation and artificial intelligence (AI). Automating data-related tasks and using AI/ML to analyze data allows businesses to reduce the time and resources needed for data management, leading to more efficient and accurate analysis. Another main key factor that will contribute to the future success of DataOps is the growing importance of data governance. As organizations collect and use more data, it is crucial to have proper controls in place to ensure data privacy/security. DataOps practices can help businesses establish and maintain effective data governance.

Overall, the future of DataOps is expected to see continued growth and evolution as businesses and organizations seek to optimize and leverage data-driven insights to drive their success.

DataOps in Action!

Previously, we discussed how Netflix uses DataOps to streamline its data management operations. To have a complete understanding of how DataOps is used in practice, let's examine a second case study. Suppose a leading online store/retailer decides to use DataOps to enhance their sales forecasting procedure. Previously, the retailer had difficulty making accurate sales forecasts due to the complexity of their data environment and the rigorous manual and laborious processes they had to go through to prepare and analyze data. To address these challenges, they formed a DataOps team that included data engineers, data scientists, and data/business analysts.

The team then implemented an automated data ingestion and transformation pipeline utilizing a market-leading data integration platform. This allowed them to swiftly and efficiently gather sales data from multiple sources, including online transactions, in-store purchases, user product preferences, user activity, and market research. The data was then cleaned, transformed, and validated using a series of predefined rules and procedures to ensure that it was ready for final analysis. The team then collaborated with data scientists to create and deploy AI/ML models that could predict future sales trends. These models were trained on historical product sales data and were designed to learn and adapt over time, becoming more accurate as more data was supplied to them. And after that, the team worked with data/business analysts to integrate the sales forecasting technique into the retailer's overall decision-making processes. This included making dashboards and reports that showed the outputs of the forecasting models and how they worked, as well as integrating the forecasts into the retailer's systems for managing product inventory and setting up product prices.

The results of the DataOps implementation were impressive. The store was able to track sales more accurately, which significantly improved managing the product inventory and aided in making smarter business decisions. Overall, the DataOps approach helped the retailer/store to make better understand and act on the data they had, leading to improved efficiency, accuracy, and agility.

Resources for learning more about DataOps

To learn more about DataOps, there are a number of resources available, including books, articles, online courses, videos and events/podcasts. Some recommendations(personal preference) includes:

Books:

• “Creating a Data-Driven Enterprise with DataOps” by Ashish Thusoo and Joydeep Sen Sarma

• "Practical DataOps: Delivering Agile Data Science at Scale" by Harvinder Atwal

• “Data Teams: A Unified Management Model for Successful Data-Focused Teams” by Jesse Anderson

• “Managing Data in Motion Data Integration Best Practice Techniques and Technologies” by April Reeve

• “The DataOps Cookbook” by Christopher Bergh

Articles:

There are many articles available online that different cover aspects of DataOps.

• Data Quality as a Crucial Part of DataOps

• A Deep Dive Into Data Quality

• Managing Data in Motion

• What is DataOps? Everything You Need to Know

• What is DataOps?

• DataOps is NOT Just DevOps for Data

• DataOps: DevOps Plus Big Data

Videos: There are several videos, online courses, and training courses available for those interested in learning more about DataOps.

• What is DataOps?

• DataOps 101 - Why, What, How?

• What is DataOps?

Conclusion

DataOps is a crucial approach to data management operations that enables businesses to improve the speed, quality, and reliability of data processing and analysis. It facilitates collaboration and communication and accelerates the delivery of insights and results at a rapid pace. While implementing DataOps can present challenges, following best practices and communicating the benefits to stakeholders can help ensure a successful adoption. As emerging technologies continue to change the industry, we may anticipate DataOps to evolve and potentially expand into more fields. Above all, DataOps is a people-driven discipline, meaning that it depends on the abilities and knowledge of the individuals. It is not a tool or application that can be bought and implemented without the required human resources. Instead, it necessitates a team of proficient data experts that can collaborate effectively and efficiently.

References

[1] Swanson, Brittany-Marie. “What is DataOps? Everything You Need to Know.” Oracle Blogs, 12 March 2018, https://blogs.oracle.com/ai-and-datascience/post/what-is-dataops-everything-you-need-to-know. Accessed 7 January 2023.

[2] DataOps and the future of data management.” MIT Technology Review, 24 September 2019, https://www.technologyreview.com/2019/09/24/132897/dataops-and-the-future-of-data-management/. Accessed 6 January 2023.

[3] Valentine, Crystal, and William Merchan. “DataOps: An Agile Methodology for Data-Driven Organizations.” Oracle, https://www.oracle.com/a/ocom/docs/oracle-ds-data-ops-map-r.pdf. Accessed 6 January 2023.

[4] Anderson, C. (2019). Creating a Data-Driven Enterprise with DataOps. O'Reilly Media, Inc. Retrieved from https://www.oreilly.com/library/view/creating-a-data-driven/9781492049227/ Accessed 6 January 2023.

[5] A. Dyck, R. Penners and H. Lichter, "Towards Definitions for Release Engineering and DevOps," 2015 IEEE/ACM 3rd International Workshop on Release Engineering.

[6] Saurabh, Saket. “What is DataOps? | Platform for the Machine Learning Age.” Nexla, https://www.nexla.com/define-dataops/. Accessed 7 January 2023.

[7] Heudecker, Nick. “Hyping DataOps - Nick Heudecker.” Gartner Blog Network, 31 July 2018, https://blogs.gartner.com/nick-heudecker/hyping-dataops/. Accessed 7 January 2023.

The topic of containerization has become increasingly important in the world of software/IT industries. Two of the most popular technologies for containerization are Docker and Containerd. While both of these technologies offer similar functionality, some key differences between them can make one better suited for your project than the other. In this article, we will explore the key differences between Docker and Containerd, and help you determine which containerization technology is right for your project.

What is Containerization?

Before we dive into the differences between Docker and Containerd, it's essential to understand what containerization is and why it has become such a popular topic in the world of software and IT industries.

In simple terms, containerization is the process of packaging an application and its dependencies into a lightweight and portable container. This container can then run on any infrastructure, whether on a local machine, in a cloud environment, or on a virtual machine.

Containers are a way to package software in a format that can run consistently on any platform. This means that a developer can create an application on their personal local machine, test it on their local machine, and then run it on a remote server, all without any changes to the code. This makes it much easier to develop and deploy applications, since there is no need to worry about platform-specific differences.

One of the key benefits of containerization is that it allows developers to create and deploy applications quickly and easily. Because the container includes everything the application needs to run, it can be deployed and run consistently on any infrastructure, without the need to worry about differences in operating systems or other dependencies. This makes it easier for developers to collaborate and share their work and for organizations to scale their applications quickly and efficiently.

What is Docker?

Docker is a fully open source containerization platform that enables developers to package and deploy their applications in a lightweight and portable manner.

Docker uses containers to create a virtual environment containing all the dependencies and libraries required to run an application. This allows developers to build, test, and deploy their applications consistently and predictably, regardless of the underlying infrastructure.

One of the main benefits of using Docker is that it allows for more efficient use of resources on a host machine. Since each container is isolated from the others, multiple containers can run on the same host without interfering with each other. This means that a single host machine can run more applications, which can be helpful for organizations that need to run a large number of applications but may not have the resources to run them all on separate machines.

Docker also makes it easier to manage and scale applications. For example, if an application needs to be scaled up to handle more traffic, it is easy to spin up additional containers to handle the extra load. This can be done quickly and easily without the need to provision and configure new machines.

Another advantage of Docker is that it allows for easier collaboration between developers. Since containers contain all of the dependencies required to run an application, it is easy for developers to share their work with others. This means that developers can work on different parts of an application simultaneously without having to worry about conflicts or compatibility issues.

What is Containerd?

Containerd also is a fully open source container runtime designed to be used as a building block for other container platforms, including Docker.

Containerd provides a low-level API that enables developers to create, manage, and distribute containers. Unlike Docker, which includes a full suite of tools and utilities for building, deploying, and managing containers, Containerd focuses on providing a simple and efficient container runtime.

At its core, containerd is a daemon that runs on a host machine and manages the lifecycle of containers. It is responsible for managing the underlying processes that make up a container and providing an interface for other tools and services to interact with those containers. This allows developers and system administrators to easily manage and deploy containers without worrying about the low-level details of how containers work.

Another important aspect of containerd is its modular design. It comprises several different components, each of which performs a specific function within the container ecosystem. This modular design allows users to customize and extend containerd to suit their particular needs easily. It also makes it easy to integrate containerd into existing systems and workflows, allowing users to leverage the power of containers without having to make significant changes to their existing infrastructure.

Key Differences Between Docker and Containerd:

Comparing containerd and Docker is like comparing a screwdriver to a toolbox. Both are essential, but they serve different purposes. While a screwdriver is a single tool used for a specific task, a toolbox holds various tools (including the screwdriver itself) that can be used for multiple tasks. Similarly, containerd is a single tool used for managing containers in a system, while Docker is a suite of tools that includes containerd and other tools for building, deploying, and running containers.

1. Architecture: One of the key differences between Docker and Containerd is the architecture of the two technologies. Docker uses a client-server architecture, where the Docker daemon (or server) is responsible for managing the containers on a host machine, and the Docker CLI (or client) is used to communicate with the daemon. Containerd, on the other hand, uses a more modular architecture, where different components can be easily added or removed to create a custom container platform.

2. Scope of functionality: Another critical difference between Docker and Containerd is the scope of functionality the two technologies provide. Docker includes a full suite of tools and utilities for building, deploying, and managing containers, including the Docker CLI, Docker Compose, and Docker Swarm. In contrast, Containerd is focused on providing a simple and efficient runtime for containers and does not include any additional tools or utilities.

3. Ease of use: Docker and Containerd also differ in terms of their ease of use. Docker is designed to be user-friendly and easy to use, with a simple command-line interface and a wealth of documentation and tutorials available online. Containerd, on the other hand, is more geared towards experienced developers and may require a deeper understanding of containerization and runtime environments to use effectively.

Which Containerization Technology is Right for Your Project?

The choice between Docker and Containerd ultimately depends on your project's specific requirements and goals. If you are looking for a complete container platform that is easy to use and provides a wide range of tools and utilities for building, deploying, and managing containers, then Docker may be the right choice. However, if you are looking for a more lightweight and modular container runtime that can be easily integrated into other container platforms, then Containerd may be a better option.

Conclusion

After comparing Docker and Containerd, it can be concluded that the right containerization technology for a project depends on that project's specific needs and requirements. Both Docker and Containerd offer useful features and capabilities for managing and deploying containers, but they differ in some key areas. Docker provides a more comprehensive and user-friendly suite of tools for building, deploying, and managing containers, while Containerd provides a more lightweight and modular option with a focus on core container runtime functionality. Ultimately, the decision on which technology to use should be based on a thorough evaluation of the project's requirements and a consideration of the strengths and limitations of each option.

What is SEO?

The technique of improving your website for search engines is known as search engine optimization (SEO). The primary goal of SEO optimization on any given webpage, web application, website, or documentation is to improve the quality of that content so that search engines will include it in their results, potentially increasing the number of visitors as well as traffics to that particular application.

How does it work?

Let's look at how SEO works in practice now that we've learned about it. So there's something known as search engine crawlers(also known as bots) which crawl through all of the content available out there on the internet, and they do so by tracking and filtering internal links within websites as well as links between websites. The crawler reads the content on each page, including redirection links, URLs, tags, headings, and a slew of other elements, to gain a sense of what the site's content is all about, and then sends that information to the search engine provider, who creates an index out of it and then ranks the site based on it.

Factors that impact SEO rankings

There are a plethora of factors that influence overall search engine rankings, for example, such as high-quality, authentic information/content, incoming links, or backlinking. These are the obvious things that might attract/lure a massive amount of user base to the content, but there are a few other important SEO factors that you might want to consider which are as follows:

• Keyword Usage
• Backlinks
• Quality Content
• Speed
• User Experience(UX)
• HTML format
• Domain authority
• Social Media Links
• Mobile Optimization

These are the factors that, when paired with the search engine's understanding of your original material, will be used to convert an index into queries/rankings for various searches conducted within search engines.

Now that we've covered the fundamentals of SEO, let's get started building an SEO-rich documentation site with Docz. To do so, follow the procedures outlined below . But first, let's have a better understanding of what docz is.

What is Docz?

Producing documentation is a very daunting task. So, docz was built to remedy this problem. Docz makes use of GatsbyJS's incredible capabilities, allowing you to swiftly construct live-reloading, SEO-friendly, production-ready, fully customisable documentation pages in just a matter of minutes. To discover everything there is to know about Docz, follow the article mentioned below.

Before we get started with docz for the purposes of this tutorial, we'll be using the gatsby theme for our docz project, so, first and foremost, let's try to understand what actually is GatsbyJS.

What is GatsbyJS?

Gatsby is a completely open-source framework that integrates the capabilities of React, GraphQL, and Webpack into a single tool for creating blazingly fast, super reliable, robust and flexible static webpages and applications.

keep in mind that docz's core is based entirely on Gatsby.

What are Gatsby Themes?

Gatsby themes are plugins that include a gatsby-config.js file and allow Gatsby sites to have pre-configured functionality, data sourcing, and/or UI code. Gatsby themes can be thought of as distinct Gatsby sites that can be put together to help users to break up a larger Gatsby project!

Docz and Gatsby

Use gatsby-theme-docz if you wish to use Docz in a Gatsby application. It's now easier than ever to get started constructing a Gatsby site, thanks to the inclusion of theming in Gatsby. As a Gatsby Theme, you can prepackage shared functionality, data sourcing, and design. Not only that, but the Docz theme includes all of the components and algorithms required to produce a documentation page, allowing users to explore and develop a truly useful documentation site.

Setting up Docz with Gatsby theme

To begin, you must first initialize npm in an empty directory, after which you can only install the packages listed below.

npm init -y

You can install it using yarn or npm.(For this tutorial I will using yarn)

npm install gatsby gatsby-theme-docz@next docz@next react react-dom

or

yarn add gatsby gatsby-theme-docz@next docz@next react react-dom

The project will take a few minutes to set up, so you can go grab a cup of coffee in the meanwhile.

docz-seo-demo> yarn add gatsby gatsby-theme-docz@next docz@next react react-dom
yarn add v1.22.4
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building packages...
...
...
...
├─ webpack@5.72.0
├─ websocket-driver@0.7.4
├─ websocket-extensions@0.1.4
├─ whatwg-url@5.0.0
├─ which-boxed-primitive@1.0.2
├─ yargs-parser@18.1.3
├─ yeast@0.1.2
├─ yn@3.1.1
└─ yocto-queue@0.1.0
Done in 573.64s.

Your package.json file should look something like this after your dependencies have been successfully installed.

{
  "name": "docz-seo-demo",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "docz": "1.0.3",
    "docz-theme-default": "1.0.3",
    "gatsby": "^2.3.23",
    "gatsby-theme-docz": "1.0.3",
    "react": "^16.8.6",
    "react-dom": "^16.8.6",
  }
}

Now that the project is up and running, create a file called gatsby-config.js and add gatsby-theme-docz to the plugins section. After you've done that, your configuration should look like this.

//gatsby-config.js
module.exports = {
  plugins: ["gatsby-theme-docz"],
};

Then, let's get to work on some documentation. To do so, simply create one mdx file and paste the following code into it.

---
name: Heelloooooo!
route: /
---

# Hello thereee!

Hello, I am awesome!!

Your file and folder structure should look something like this

Now just run gatsby in development mode:

yarn gatsby develop

It's time to make a docz configuration file, so make a file called doczrc.js and paste the following code into it.

 module.exports = {
  plugins: [
    {
      resolve: "gatsby-theme-docz",
    },
  ],
};

Use options inside the plugin definition to set some settings for your theme.

//doczrc.js
module.exports = {
  plugins: [
    {
      resolve: 'gatsby-theme-docz',
      options: {
        /* ADD your custom options */
      },
    },
  ],
}

Let's make our Documentation SEO friendly

To make our documentation SEO-friendly, we must first include metadata, a favicon, and a description. To do so, follow the procedures given below.

Configuring Gatsby in conjunction with react-helmet-async is how you add metadata to your site. Keep in mind that we're talking about react-helmet-async, not react-helmet. Because react-helmet-async is an API-compatible clone, you should only need to import from a different package.

Component shadowing

Component Shadowing is a concept that Gatsby Themes introduces. This feature enables users to alter the rendering of a component by overriding it. Other theming systems make it impossible to change features of a theme if there isn't a built-in configuration option. Component Shadowing is a powerful escape hatch for users who want to make rapid, one-time modifications. If the component that you want to shadow has the matching folder structure and name the theme component will get automatically shadowed.

Overriding the wrapper component

The metadata is configured in a file called wrapper.js in the gatsby-theme-docz docz theme package. To override it, we must Shadow the component, which means we must create a copy of the file in our own source directory with the same file path and its name.

Now paste the following code into the wrapper.js file you just created. This component should be used as a wrapper instead of the main core theme component. So, this wrapper consists of all the necessary elements that are required to make the documentation SEO rich.

// wrapper.js
import * as React from 'react'
import { Helmet } from 'react-helmet-async'

const Wrapper = ({ children, doc }) => <React.Fragment>
    <Helmet>
        <meta charSet="utf-8" />
        <title>My Shadow!</title>
        <link rel="icon"
            type="image/png"
              href="http://example.com/myicon.png"
        />
    </Helmet>
    {children}
</React.Fragment>
export default Wrapper

The next stage in making your documentation SEO friendly is to provide very specific descriptions, titles, and keywords, for example.

---
title: Very detailed title
description: Short description of the page
keywords: [Perfect, Matching, Keywords]
---

Static HTML generation

Docz with gatsby works as a static site generator, which means that HTML files are generated statically for each URL route, making it easier for search engines to find your material/content.

Image alt description​

When an image can't be seen visually, such as when using a screen reader or when the image fails t load, the alt tag tells the search engine what the image is all about. In Markdown, alt tags are also supported. You can also give your image a title, which has a huge impact on SEO but is displayed as a tooltip when hovering over it, and is typically used to provide info of that particular image.

![This is an image of something](./image/image.png)

Readable links​

Docz creates links from your file names, but you may readily modify that using route according to your own desire.

Structured content​

To understand the structure of your webpage, search engines rely on HTML code. When Docz displays your pages, semantic markup is utilized to divide the various portions of the page, assisting the search engine in locating parts and its different content.

Conclusion

To summarize what we learned, we learnt about SEO, then we learned about docz, and we configured and implemented docz with a custom gatsby theme, and finally, we specified completely on the factors that affect the SEO of the documentation by applying and demonstrating by example.

• What is Typesense?

This article will walk you through installing Typesense, creating a new application from scratch, preconfiguring the client, and more. This article will also show you how to construct a Typesense Collection. Finally, we’ll start our app and initialize it, then add an item to our collection and start searching.

Building a Song Search Application from scratch!

Typesense Configuration

Typesense can be used by installing its prebuilt docker image which is already provided to us by Typesense or by using the Typesense cloud hosting solution, which is the most straightforward way to get started. To get started, go to the Typesense cloud website and sign up with your Github account, or use the docker method easily and directly. We’ll use the Docker method for the purposes of this tutorial.To follow along with this tutorial, go to the Typesense Dockerhub and download the prebuilt docker image, then follow the instructions below.

docker pull typesense/typesense

and

mkdir /tmp/typesense-data
docker run -p 8108:8108 -v/tmp/data:/data typesense/typesense:0.22.2 --data-dir /data --api-key=songsearch --enable-cors --listen-port 8108

Building Search UIs from scratch.

Using the open source InstantSearch.js library library or its React, Vue, and Angularjs, as well as the Typesense-InstantSearch-Adapter, you can create a plug-and-play full-featured search interface with just a few lines of code.

Finally, lets’s actually build a search UI for our Application .

Step-by-step instructions for installing

Let’s begin with a basic template:

npx create-instantsearch-app typesense-songsearch

? The name of the application or widget typesense-songsearch
? InstantSearch template InstantSearch.js

Creating a new InstantSearch app in typesense-songsearch.

? InstantSearch.js version 4.38.0
? Application ID latency
? Search API key 6be0576ff61c053d5f9a3225e2a90f76
? Index name instant_search
? Attributes to display
  Used to generate the default result template
? Attributes to display
  Used to filter the search interface Dynamic widgets

📦  Installing dependencies...

yarn install v1.22.0
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 🔨  Building fresh packages...
success Saved lockfile.

Done in 173.57s.

🎉  Created typesense-songsearch at typesense-songsearch.

Begin by typing:

  cd typesense-songsearch
  yarn start

⚡️  Start building something awesome!

This is how your folder structure should look after you’ve finished installing your Instantsearch app (UI).

For the npx create-instantsearch-app command, here are some setup suggestions: Any of the web libraries supported by typesense can be used, including InstantSearch.js, React, Vue, and Angular. The default version of InstantSearch.js can be used. The application ID can be any string; we’ll be changing it later in the article. Search API key: any string - later in the guide, we’ll replace this with the Typesense Search-only API Key and Our index name is the name of the collection in Typesense, and lastly , leave the Displaying attributes as it is.

Next, we’ll need to install the Typesense InstantSearch adapter to use InstantSearch with a Typesense backend:

npm install typesense-instantsearch-adapter

Installing Typesense

npm install typesense

We’re ready to create a Typesense collection, Now that everything is installed and running, we can create a Typesense collection, index some documents in it, and try searching for them.

Your package.json file should look like this after you’ve installed all of the necessary dependencies.

{
  "name": "typesense-songsearch",
  "version": "1.0.0",
  "private": true,
  "main": "src/app.js",
  "scripts": {
    "start": "parcel index.html --port 3000",
    "build": "parcel build index.html",
    "lint": "eslint .",
    "lint:fix": "npm run lint -- --fix"
  },
  "devDependencies": {
    "babel-eslint": "10.0.3",
    "eslint": "5.7.0",
    "eslint-config-algolia": "13.2.3",
    "eslint-config-prettier": "3.6.0",
    "eslint-plugin-import": "2.19.1",
    "eslint-plugin-prettier": "3.1.2",
    "parcel-bundler": "1.12.5",
    "prettier": "1.19.1"
  },
  "dependencies": {
    "algoliasearch": "4",
    "instantsearch.js": "4.38.0",
    "typesense": "^0.14.0",
    "typesense-instantsearch-adapter": "^2.2.0"
  }
}

As you can see, it’s a very basic app with only two important files: app.js and index.html, which contains the structure of our entire app.

In the index.html file, you can see an id. We can use those ids to attach UI components such as a searchbox, results, and facets if you want to include them, as well as pagination widgets.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1, shrink-to-fit=no"
    />
    <meta name="theme-color" content="#000000" />

    <link rel="manifest" href="./manifest.webmanifest" />
    <link rel="shortcut icon" href="./favicon.png" />

    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/npm/instantsearch.css@7/themes/algolia-min.css"
    />
    <link rel="stylesheet" href="./src/index.css" />
    <link rel="stylesheet" href="./src/app.css" />

    <title>typesense-songsearch</title>
  </head>

  <body>
    <header class="header">
      <h1 class="header-title">
        <a href="/">typesense-songsearch</a>
      </h1>
      <p class="header-subtitle">
        using
        <a href="https://github.com/algolia/instantsearch.js">
          InstantSearch.js
        </a>
      </p>
    </header>

    <div class="container">
      <div class="search-panel">
        <div class="search-panel__filters">
          <div id="dynamic-widgets"></div>
        </div>

        <div class="search-panel__results">
          <div id="searchbox"></div>
          <div id="hits"></div>
        </div>
      </div>

      <div id="pagination"></div>
    </div>

    <script src="https://cdn.jsdelivr.net/npm/algoliasearch@4.10.5/dist/algoliasearch-lite.umd.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/instantsearch.js@4.38.0"></script>
    <script src="./src/app.js"></script>
  </body>
</html>

As you can see below, the app.js file is configured for algolia, and we’ll need to change that for typesense.

// app.js
const { algoliasearch, instantsearch } = window;

const searchClient = algoliasearch("latency", "songsearch");

const search = instantsearch({
  indexName: "instant_search",
  searchClient,
});
search.addWidgets([
  instantsearch.widgets.searchBox({
    container: "#searchbox",
  }),
  instantsearch.widgets.hits({
    container: "#hits",
  }),
  instantsearch.widgets.configure({
    facets: ["*"],
    maxValuesPerFacet: 20,
  }),
  instantsearch.widgets.dynamicWidgets({
    container: "#dynamic-widgets",
    fallbackWidget({ container, attribute }) {
      return instantsearch.widgets.refinementList({
        container,
        attribute,
      });
    },
    widgets: [],
  }),
  instantsearch.widgets.pagination({
    container: "#pagination",
  }),
]);

search.start();

To use InstantSearch with a Typesense backend, we’ll need to install the Typesense InstantSearch adapter. To do so, simply open a terminal window and type the following command into it.

If you’ve already installed it, you can skip this step.

npm install --save typesense-instantsearch-adapter

Finally, the project can be adjusted to use Typesense. In the background, we’ve got our typesense instance up and running. Open src/app.js and change the way InstantSearch is initialized to get InstantSearch.js to use the Typesense adapter:

const searchClient = algoliasearch("latency", "songsearch");

const search = instantsearch({
  indexName: "instant_search",
  searchClient,
});

to this

const typesenseInstantsearchAdapter = new TypesenseInstantSearchAdapter({
  server: {
    apiKey: "songsearch",
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
  },
  additionalSearchParameters: {
    queryBy: "title,primary_artist_name,album_name", //quering by
  },
});
const searchClient = typesenseInstantsearchAdapter.searchClient;

const search = instantsearch({
  searchClient,
  indexName: "songs",
});

Now we need to import our songs data, so create a dataset folder and inside it, create your own songs json file, filling it with all of the necessary song information, or download the songs dataset from here. Finally, your folder structure should look something like this.

Let’s get to work on the data-importing scripts. We’ll start by creating a file called loadData.js and we will export and initialize the typesense client inside but before we do that we need to install a few packages lodash and fast-json-stringify.

npm i lodash

and

npm i fast-json-stringify

So, at the very top, import all of the dependencies that this script requires to run, as shown below.

// loadData.js
const _ = require("lodash");
const fastJson = require("fast-json-stringify");
const fs = require("fs");
const readline = require("readline");
const Typesense = require("typesense");

After that, Create a stringify based on the schema of the documents that need to be stringified.

// loadData.js
const _ = require("lodash");
const fastJson = require("fast-json-stringify");
const fs = require("fs");
const readline = require("readline");
const Typesense = require("typesense");

const stringify = fastJson({
  title: "Song Schema",
  type: "object",
  properties: {
    track_id: {
      type: "string",
    },
    title: {
      type: "string",
    },
    album_name: {
      type: "string",
      nullable: true,
    },
    primary_artist_name: {
      type: "string",
    },
    genres: {
      type: "array",
      items: {
        type: "string",
      },
    },
    country: {
      type: "string",
    },
    release_date: {
      type: "integer",
    },
    release_decade: {
      type: "string",
    },
    release_group_types: {
      type: "array",
      items: {
        type: "string",
      },
    },
    urls: {
      type: "array",
      items: {
        type: "object",
        properties: {
          type: {
            type: "string",
          },
          url: {
            type: "string",
          },
        },
      },
    },
  },
});

Create a new extractUrls function.

function extractUrls(parsedRecord) {
  return parsedRecord["relations"]
    .filter((r) =>
      [
        "amazon asin",
        "streaming",
        "free streaming",
        "download for free",
        "purchase for download",
      ].includes(r["type"])
    )
    .map((r) => {
      return { type: r["type"], url: r["url"]["resource"] };
    });
}

Now create an asynchronous addSongsToTypesense function and copy the code exactly as it is shown below. This function will simply index all of the songs into Typesense, and if an error occurs during the process, it will simply log an error.

async function addSongsToTypesense(songs, typesense, songsCollection) {
  try {
    const returnDataChunks = await Promise.all(
      _.chunk(songs, Math.ceil(songs.length / CHUNK_SIZE)).map((songsChunk) => {
        const jsonlString = songsChunk
          .map((song) => stringify(song))
          .join("\n");

        return typesense
          .collections(songsCollection)
          .documents()
          .import(jsonlString);
      })
    );

    const failedItems = returnDataChunks
      .map((returnData) =>
        returnData
          .split("\n")
          .map((r) => JSON.parse(r))
          .filter((item) => item.success === false)
      )
      .flat();
    if (failedItems.length > 0) {
      throw new Error(
        `Error indexing items ${JSON.stringify(failedItems, null, 2)}`
      );
    }
  } catch (error) {
    console.log(error);
  }
}

A Collection in Typesense is a set of related Documents that functions similarly to a table in a relational database. We give a collection a name and describe the fields that will be indexed when a document is added to the collection when we create it.

Now go to the loadData.js file and add the following changes to the code.

module.exports = (async () => {
  const typesense = new Typesense.Client({
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "songsearch",
  });

  const songsCollection = "songs";
  const schema = {
    name: songsCollection,
    fields: [
      { name: "track_id", type: "string" },
      { name: "title", type: "string" },
      { name: "album_name", type: "string", optional: true },
      { name: "primary_artist_name", type: "string", facet: true },
      { name: "genres", type: "string[]", facet: true },
      { name: "country", type: "string", facet: true },
      { name: "release_date", type: "int64" },
      { name: "release_decade", type: "string", facet: true },
      { name: "release_group_types", type: "string[]", facet: true },
    ],
    default_sorting_field: "release_date",
  };

  console.log(`Populating data in Typesense "${songsCollection}" collection`);

  console.log(`Creating schema...`);
  await typesense.collections().create(schema);

  console.log(`Songs records...`);

  const fileStream = fs.createReadStream(DATA_FILE);
  const rl = readline.createInterface({
    input: fileStream,
    crlfDelay: Infinity,
  });

  let songs = [];
  let currentLine = 0;
  for await (const line of rl) {
    currentLine += 1;
    const parsedRecord = JSON.parse(line);
    try {
      songs.push(
        ...parsedRecord["media"]
          .map((media) => media["tracks"])
          .flat()
          .filter((track) => track) // To remove nulls
          .map((track) => {
            const releaseDate =
              Math.round(
                Date.parse(
                  parsedRecord["release-group"]["first-release-date"]
                ) / 1000
              ) || 0;

            // Be sure to update the schema passed to stringify when updating this structure
            const song = {
              track_id: track["id"],
              title: track["title"],
              album_name: parsedRecord["title"],
              primary_artist_name:
                parsedRecord["artist-credit"][0]["artist"]["name"],
              genres: [
                ...track["recording"]["genres"].map((g) => g.name),
                ...parsedRecord["genres"].map((g) => g.name),
                ...parsedRecord["release-group"]["genres"].map((g) => g.name),
              ].map(
                ([firstChar, ...rest]) =>
                  firstChar.toUpperCase() + rest.join("").toLowerCase()
              ),
              country: parsedRecord["country"] || "Unknown",
              release_date: releaseDate,
              release_decade: `${
                Math.round(new Date(releaseDate * 1000).getUTCFullYear() / 10) *
                10
              }s`,
              release_group_types: [
                parsedRecord["release-group"]["primary-type"] || "Unknown",
                parsedRecord["release-group"]["secondary-types"] || null,
              ]
                .flat()
                .filter((e) => e),
              urls: extractUrls(parsedRecord),
            };
            process.stdout.write("-");

            return song;
          })
      );
    } catch (e) {
      console.error(e);
      console.error(parsedRecord);
      throw e;
    }

    if (currentLine >= MAX_LINES) {
      break;
    }
  }

  if (songs.length > 0) {
    await addSongsToTypesense(songs, typesense, songsCollection);
    console.log("ALL songs INDEXED ✅ !");
  }
})();

If an error occurs while loading the data, simply add the following snippet of code to the loadData.js file before creating songSchema.

await typesense.collections("songs").delete();

This is how your final code should look like.

// loadData.js
const _ = require("lodash");
const fastJson = require("fast-json-stringify");
const fs = require("fs");
const readline = require("readline");
const Typesense = require("typesense");

const stringify = fastJson({
  title: "Song Schema",
  type: "object",
  properties: {
    track_id: {
      type: "string",
    },
    title: {
      type: "string",
    },
    album_name: {
      type: "string",
      nullable: true,
    },
    primary_artist_name: {
      type: "string",
    },
    genres: {
      type: "array",
      items: {
        type: "string",
      },
    },
    country: {
      type: "string",
    },
    release_date: {
      type: "integer",
    },
    release_decade: {
      type: "string",
    },
    release_group_types: {
      type: "array",
      items: {
        type: "string",
      },
    },
    urls: {
      type: "array",
      items: {
        type: "object",
        properties: {
          type: {
            type: "string",
          },
          url: {
            type: "string",
          },
        },
      },
    },
  },
});

const BATCH_SIZE = 200;
const CHUNK_SIZE = 3;
const MAX_LINES = Infinity;
const DATA_FILE = "./dataset/songs.json";

function extractUrls(parsedRecord) {
  return parsedRecord["relations"]
    .filter((r) =>
      [
        "amazon asin",
        "streaming",
        "free streaming",
        "download for free",
        "purchase for download",
      ].includes(r["type"])
    )
    .map((r) => {
      return { type: r["type"], url: r["url"]["resource"] };
    });
}

async function addSongsToTypesense(songs, typesense, songsCollection) {
  try {
    const returnDataChunks = await Promise.all(
      _.chunk(songs, Math.ceil(songs.length / CHUNK_SIZE)).map((songsChunk) => {
        const jsonlString = songsChunk
          .map((song) => stringify(song))
          .join("\n");

        return typesense
          .collections(songsCollection)
          .documents()
          .import(jsonlString);
      })
    );

    const failedItems = returnDataChunks
      .map((returnData) =>
        returnData
          .split("\n")
          .map((r) => JSON.parse(r))
          .filter((item) => item.success === false)
      )
      .flat();
    if (failedItems.length > 0) {
      throw new Error(
        `Error indexing items ${JSON.stringify(failedItems, null, 2)}`
      );
    }
  } catch (error) {
    console.log(error);
  }
}

module.exports = (async () => {
  const typesense = new Typesense.Client({
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "songsearch",
  });

  await typesense.collections("songs").delete();

  const songsCollection = "songs";
  const schema = {
    name: songsCollection,
    fields: [
      { name: "track_id", type: "string" },
      { name: "title", type: "string" },
      { name: "album_name", type: "string", optional: true },
      { name: "primary_artist_name", type: "string", facet: true },
      { name: "genres", type: "string[]", facet: true },
      { name: "country", type: "string", facet: true },
      { name: "release_date", type: "int64" },
      { name: "release_decade", type: "string", facet: true },
      { name: "release_group_types", type: "string[]", facet: true },
    ],
    default_sorting_field: "release_date",
  };

  console.log(`Populating data in Typesense "${songsCollection}" collection`);

  console.log(`Creating schema...`);
  await typesense.collections().create(schema);

  console.log(`Songs records...`);

  const fileStream = fs.createReadStream(DATA_FILE);
  const rl = readline.createInterface({
    input: fileStream,
    crlfDelay: Infinity,
  });

  let songs = [];
  let currentLine = 0;
  for await (const line of rl) {
    currentLine += 1;
    const parsedRecord = JSON.parse(line);
    try {
      songs.push(
        ...parsedRecord["media"]
          .map((media) => media["tracks"])
          .flat()
          .filter((track) => track) // To remove nulls
          .map((track) => {
            const releaseDate =
              Math.round(
                Date.parse(
                  parsedRecord["release-group"]["first-release-date"]
                ) / 1000
              ) || 0;

            // Be sure to update the schema passed to stringify when updating this structure
            const song = {
              track_id: track["id"],
              title: track["title"],
              album_name: parsedRecord["title"],
              primary_artist_name:
                parsedRecord["artist-credit"][0]["artist"]["name"],
              genres: [
                ...track["recording"]["genres"].map((g) => g.name),
                ...parsedRecord["genres"].map((g) => g.name),
                ...parsedRecord["release-group"]["genres"].map((g) => g.name),
              ].map(
                ([firstChar, ...rest]) =>
                  firstChar.toUpperCase() + rest.join("").toLowerCase()
              ),
              country: parsedRecord["country"] || "Unknown",
              release_date: releaseDate,
              release_decade: `${
                Math.round(new Date(releaseDate * 1000).getUTCFullYear() / 10) *
                10
              }s`,
              release_group_types: [
                parsedRecord["release-group"]["primary-type"] || "Unknown",
                parsedRecord["release-group"]["secondary-types"] || null,
              ]
                .flat()
                .filter((e) => e),
              urls: extractUrls(parsedRecord),
            };
            process.stdout.write("-");

            return song;
          })
      );
    } catch (e) {
      console.error(e);
      console.error(parsedRecord);
      throw e;
    }

    if (currentLine >= MAX_LINES) {
      break;
    }
  }

  if (songs.length > 0) {
    await addSongsToTypesense(songs, typesense, songsCollection);
    console.log("ALL songs INDEXED ✅ !");
  }
})();

Finally, type node loadData.js into the terminal of that same project directory, and you have successfully imported 1 thousand songs documents into the search index.

Now, you can start your application by typing npm start in the terminal of that project directory.

We need to fix the fact that the application functions perfectly but no information, as well as data, is displayed in it.

So, to fix it, go to our app.js file and remove a few of the widgets that we don’t need, so the app.js file should look something like this.

// app.js
const { instantsearch } = window;

import TypesenseInstantSearchAdapter from "typesense-instantsearch-adapter";

const typesenseInstantsearchAdapter = new TypesenseInstantSearchAdapter({
  server: {
    apiKey: "songsearch",
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
  },
  additionalSearchParameters: {
    queryBy: "title,primary_artist_name,album_name", //quering by
  },
});
const searchClient = typesenseInstantsearchAdapter.searchClient;

const search = instantsearch({
  searchClient,
  indexName: "songs",
});

search.addWidgets([
  instantsearch.widgets.searchBox({
    container: "#searchbox",
  }),
  instantsearch.widgets.hits({
    container: "#hits",
  }),
  instantsearch.widgets.pagination({
    container: "#pagination",
  }),
]);

search.start();

Simply re-run the application after you’ve fixed it, and your application should now look like this.

You can now build a search interface using any of the InstantSearch widgets, and we’ll be adding a template to present the data in a pleasing manner.Simplycopy the template code below and add it inside the #hits widget present inside app.js file.

templates: {
item: `
<h2>
{ "attribute": "title" }
</h2>
<div>
by
<a style="color:#FF6F82">{ "attribute": "primary_artist_name" }</a>
</div>
<div>
Album: <a style="color:#8661d1" >{ "attribute": "album_name" }</a>
</div>
<div class="text-muted small mb-2">
</div> `,
},

app.js file should resemble something like this.

// App.js
const { instantsearch } = window;

import TypesenseInstantSearchAdapter from 'typesense-instantsearch-adapter';

const typesenseInstantsearchAdapter = new TypesenseInstantSearchAdapter({
  server: {
    apiKey: 'songsearch',
    nodes: [
      {
        host: 'localhost',
        port: '8108',
        protocol: 'http',
      },
    ],
  },
  additionalSearchParameters: {
    queryBy: 'title,primary_artist_name,album_name', //quering by
  },
});
const searchClient = typesenseInstantsearchAdapter.searchClient;

const search = instantsearch({
  searchClient,
  indexName: 'songs',
});

search.addWidgets([
  instantsearch.widgets.searchBox({
    container: '#searchbox',
  }),
  instantsearch.widgets.hits({
    container: '#hits',
    templates: {
      item: `
      <h2>
        { "attribute": "title" }
      </h2>
      <div>
        by
        <a style="color:#FF6F82">{ "attribute": "primary_artist_name" }</a>
      </div>
      <div>
        Album: <a style="color:#8661d1" >{ "attribute": "album_name" }</a>
      </div>
      <div class="text-muted small mb-2">

      </div>
      `,
    },
  }),
  instantsearch.widgets.pagination({
    container: '#pagination',
  }),
]);

search.start();

After you’ve fixed it, re-run the application, and your app should now look like this.

Simply add the following code above the #hits widgets inside the app.js file to configure the 20 hits per page or to configure to your own requirements.

instantsearch.widgets.configure({
    hitsPerPage: 20,
  }),

So now let’s try to add the facets, to do that we’ll add an instant search widgets .In the instantsearch library this is called refinementlist and finally we have to specify the container, in our case let’s call it a #genreFilter,#artistFilter and then the attribute we want to take, in our case let’s call it genres and primary_artist_name respectively. So, our final code should look like this.

// app.js
const { instantsearch } = window;

import TypesenseInstantSearchAdapter from "typesense-instantsearch-adapter";

const typesenseInstantsearchAdapter = new TypesenseInstantSearchAdapter({
  server: {
    apiKey: "songsearch",
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
  },
  additionalSearchParameters: {
    queryBy: "title,primary_artist_name,album_name", //quering by
  },
});
const searchClient = typesenseInstantsearchAdapter.searchClient;

const search = instantsearch({
  searchClient,
  indexName: "songs",
});

search.addWidgets([
  instantsearch.widgets.searchBox({
    container: "#searchbox",
  }),
  instantsearch.widgets.configure({
    hitsPerPage: 20,
  }),
  instantsearch.widgets.hits({
    container: "#hits",
    templates: {
      item: `
      <h2>
        { "attribute": "title" }
      </h2>
      <div>
        by
        <a style="color:#FF6F82">{ "attribute": "primary_artist_name" }</a>
      </div>
      <div>
        Album: <a style="color:#8661d1" >{ "attribute": "album_name" }</a>
      </div>
      <div class="text-muted small mb-2">

      </div>
      `,
    },
  }),
  instantsearch.widgets.refinementList({
    container: "#genreFilter",
    attribute: "genres",
    searchable: true,
    searchablePlaceholder: "Search genres",
    showMore: true,
  }),
  instantsearch.widgets.refinementList({
    container: "#artistFilter",
    attribute: "primary_artist_name",
    searchable: true,
    searchablePlaceholder: "Search Artist",
    showMore: true,
  }),
  instantsearch.widgets.pagination({
    container: "#pagination",
  }),
]);

search.start();

and our index.html file should look something like this.

<!-- index.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1, shrink-to-fit=no"
    />
    <meta name="theme-color" content="#000000" />

    <link rel="manifest" href="./manifest.webmanifest" />
    <link rel="shortcut icon" href="./favicon.png" />

    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/npm/instantsearch.css@7/themes/algolia-min.css"
    />
    <link rel="stylesheet" href="./src/index.css" />
    <link rel="stylesheet" href="./src/app.css" />

    <title>typesense-songsearch</title>
  </head>

  <body>
    <header class="header">
      <h1 class="header-title">
        <a href="/">typesense-songsearch</a>
      </h1>
      <p class="header-subtitle">
        using
        <a href="https://github.com/algolia/instantsearch.js">
          InstantSearch.js
        </a>
      </p>
    </header>

    <div class="container">
      <div class="search-panel">
        <div class="search-panel__filters">
          <div id="genreFilter"></div>
          <br />
          <div id="artistFilter"></div>
        </div>

        <div class="search-panel__results">
          <div id="searchbox"></div>
          <div id="hits"></div>
        </div>
      </div>

      <div id="pagination"></div>
    </div>

    <script src="https://cdn.jsdelivr.net/npm/algoliasearch@4.10.5/dist/algoliasearch-lite.umd.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/instantsearch.js@4.38.0"></script>
    <script src="./src/app.js"></script>
  </body>
</html>

This is how your application should look like.

Finally, we can add sorting functionality to the app by repeating the exact steps above: add the sortBy widget, specify the container with #sort-songs, specify the items with the label default with the value songs, and then create another label called Recent songs with the value songs/sort/release_date:asc, and another label called Oldest songs with the value songs/sort/release_date:dsc

 instantsearch.widgets.sortBy({
    container: '#sort-songs',
    items: [
      { label: 'Default', value: `songs` },
      { label: 'Recent songs', value: `songs/sort/release_date:asc` },
      { label: 'Oldest songs', value: `songs/sort/release_date:desc` },
    ],
  }),

Finally, update and add sort-songs id before the searchbox inside index.html file.

<!-- index.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1, shrink-to-fit=no"
    />
    <meta name="theme-color" content="#000000" />

    <link rel="manifest" href="./manifest.webmanifest" />
    <link rel="shortcut icon" href="./favicon.png" />

    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/npm/instantsearch.css@7/themes/algolia-min.css"
    />
    <link rel="stylesheet" href="./src/index.css" />
    <link rel="stylesheet" href="./src/app.css" />

    <title>typesense-songsearch</title>
  </head>

  <body>
    <header class="header">
      <h1 class="header-title">
        <a href="/">typesense-songsearch</a>
      </h1>
      <p class="header-subtitle">
        using
        <a href="https://github.com/algolia/instantsearch.js">
          InstantSearch.js
        </a>
      </p>
    </header>

    <div class="container">
      <div class="search-panel">
        <div class="search-panel__filters">
          <div id="genreFilter"></div>
          <br />
          <div id="artistFilter"></div>
        </div>

        <div class="search-panel__results">
          <div id="sort-songs"></div>
          <div id="searchbox"></div>
          <div id="hits"></div>
        </div>
      </div>

      <div id="pagination"></div>
    </div>

    <script src="https://cdn.jsdelivr.net/npm/algoliasearch@4.10.5/dist/algoliasearch-lite.umd.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/instantsearch.js@4.38.0"></script>
    <script src="./src/app.js"></script>
  </body>
</html>

Let’s take a look at the finished product of our typesense-integrated songsearch application.

The entire source code of the application can be found here

Typesense was built with several distinctive features primarily aimed at making the developer’s job easier while also giving customer as well as user the ability to provide a better search experience as possible.This article may have been entertaining as well as instructive in terms of how to create a fullstack song search app using typesense from the ground. Join Aviyel’s community to learn more about the open source project, get tips on how to contribute, and join active dev groups.

Aviyel is a collaborative platform that assists open source project communities in monetizing and long-term sustainability. To know more visit Aviyel.com and find great blogs and events, just like this one! Sign up now for early access, and don’t forget to follow us on our socials!

This tutorial will show you how to install Typesense, how to create a Typesense application from the ground up, how to preconfigure the Typesense client, and much more. This article also demonstrates how to establish a Typesense Collection. Finally, we'll start up our program, add a new item to our collection, and search our indexed data/collection.

Let's get started. The goal of this article is to create an instant search type application, also known as "search as you type," which means that whenever you type something, the results appear instantly, providing a pleasant user experience. So, in a previous article, we created a simple Typesense Booksearch javascript application, and in this article, we'll built an Anime search but by using Animes dataset and also by using react.js, with the goal of simply showing you how to do it using the most popular framework or UI library.So let's get started with our React js application with Javascript. To do so, simply follow the instructions below.

Configuring our React application

We'll begin by using create-react-app to set up our frontend. We'll build the user interface and its features from the ground up. Let's go to work on our application right away.

Setting up react application bootstarpped using CRA

Let's start with the react part and start building it. The first thing you need do is install Node.js if it isn't already installed on your PC. So, head over to the official Node.js website and download the latest version. Node js is required in order to use the node package manager, generally known as npm. Now open the folder in your preferred code editor. we'll be using VScode code editor for this article tutorial . Next, open the integrated terminal and type npx create-react-app . This command will create a react application in the current directory.

It normally only takes a few minutes to set up. Normally, we would use npm to get packages into a project, but in this case, we'll use npx, the package runner, which will download and configure everything for us so that we can get started with an excellent template right away. It's time to start our development server, so run npm start and the browser will open react-app instantly.

So, this is how the boilerplate template appears right away. Now it's time to investigate the file and folder structure provided by create-react-app. There is a folder called node module that contains all of our node dependencies. Then there's a public folder, where the only thing that matters is the index.html file. So this appears to be a standard HTML file, complete with head, body, and meta tags. You'll notice a div with the id root inside our body tag, followed by the fallback noscript tag, which will be visible only if the user's browser has javascript disabled.

<!--index.html-->

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="theme-color" content="#000000" />
    <meta
      name="description"
      content="Web site created using create-react-app"
    />
    <link rel="apple-touch-icon" href="%PUBLIC_URL%/logo192.png" />
    <link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
    <title>React practice</title>
  </head>
  <body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
  </body>
</html>

So you're probably wondering where the content comes from. Remember that, All of our source code is contained within our source or src folder, and react will inject it into the root div element. Let's take a look at our src folder, which contains some stylesheets, javascript files, and SVG files.

Now, head over to our App.js file

// App.js

import logo from "./logo.svg";
import "./App.css";

function App() {
  return (
    <div className="App">
      <header className="App-header">
        <img src={logo} className="App-logo" alt="logo" />
        <p>
          Edit <code>src/App.js</code> and save to reload.
        </p>
        <a
          className="App-link"
          href="https://reactjs.org"
          target="_blank"
          rel="noopener noreferrer"
        >
          Learn React
        </a>
      </header>
    </div>
  );
}
export default App;

In this case, we're simply importing react from react and logo from our logo using standard javascript. Following that, we have a normal javascript function called APP, and this function in react is known as a functional component, and this function is returning a react-element that looks like HTML but is actually an jsx as you can see there is a div tag with a className of APP, and we can't say class by itself because the class is a reserved word in javascript, so in jsx we have to use className. Following that, we have the header and then the image, and notice on the image source that we have our logo, which is actually a javascript variable that we imported at the top, so in order to use the javascript within JSX, we must surround it with curly brackets, and then we have a paragraph, an anchor tag, and that is all for this component.

NOTE: Because of the export, we are able to extract the component and place it on the webpage. Export appears at the bottom of the app.js file, indicating that we are exporting the App function.

So, Now let's look at the index.js file.

// index.js

import React from "react";
import ReactDOM from "react-dom";
import "./index.css";
import App from "./App";
import reportWebVitals from "./reportWebVitals";

ReactDOM.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>,
  document.getElementById("root")
);
reportWebVitals();

So, in this case, we're importing react from react again, and this time we're also importing react-dom, and then we're importing the CSS stylesheet file, and finally, we're importing App from App.js, which is the file we just discussed, and there's service worker, which is used to make your application work completely offline. Then we invoke ReactDom.render, which accepts two parameters. The first parameter is the jsx object, and within jsx we can include our user-defined components, so react strict mode is a react defined component, whereas App is a user-defined component, and the second parameter is document.getElementById('root'), which targets the root div in our index.html file and is how we access the content in our webpage.

Note: ReactDom renders our content into our root div located at our index.html file.

React boilerplate files cleanup

We must first tidy up our projects by eliminating some of the files provided by create-react-app before we can begin creating them. After you've cleaned up your files and folder , they should look like this.

Adding and Installing some packages

We will need to install a few third-party packages for this project. so copy and paste the following command into your terminal

Installing typesense

This is going to be our primary typesense package.

npm install typesense

Installing typesense-instantsearch-adapter

This package will allow us to use the instantsearch UI, and this adapter will basically plug it into typesense because instantsearch.js is created by algolia, but typesense created this adapter to port its functionality and features into the typesense package itself.

npm install typesense-instantsearch-adapter

Installing styled-components

This package will lets you write actual CSS inside your JavaScript project.

npm install styled-components

Installing instantsearch.css

Instead of creating everything from scratch, this package will give pre-made styles such as the search box and numerous UI styles.

npm install instantsearch.css

Installing react-instantsearch-dom

This package is the React version of Algolia's instantsearch.js library which will provides us the components that we need render it onto our projects.

This package wil

npm install react-instantsearch-dom

Finally, your package.json file should look like this once you've installed all of your project's dependencies.

{
  "name": "anime-searchapp-reactjs-typesense",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@testing-library/jest-dom": "^5.14.1",
    "@testing-library/react": "^12.0.0",
    "@testing-library/user-event": "^13.2.1",
    "instantsearch.css": "^7.4.5",
    "react": "^17.0.2",
    "react-dom": "^17.0.2",
    "react-instantsearch-dom": "^6.22.0",
    "react-scripts": "5.0.0",
    "styled-components": "^5.3.3",
    "typesense": "^1.1.3",
    "typesense-instantsearch-adapter": "^2.3.0",
    "web-vitals": "^2.1.0"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": ["react-app", "react-app/jest"]
  },
  "browserslist": {
    "production": [">0.2%", "not dead", "not op_mini all"],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

Now that our project and dependencies have been set up, we can get started.Firstly, we need to import our animes data, so create a dataset folder and inside it, create your own anime list json file, filling it with all of the necessary product information, or download the anime dataset from here. Finally, your folder structure and dataset should look something like this.

So, in order to populate our Typesense index, we must first start our Typesense server, and there are several ways to do so. There is the old programmatic way, which requires you to install everything, start it up, and provide the configuration using code, or there is this one awesome method, which is the preferred method, which is very steady and known to work, and which is used in the docker image and run the docker container that will basically bind and give it a volume that will connect to where you'll be storing the data, and that's all there is to it.So that's essentially what we'll be using in this article.

Let's start by creating a new script and a folder called scripts, because we'll have a handful of scripts loaded in here that will allow us to either index the data or start the typesense server or the docker container, so you can basically put it inside the script run tag inside the package.json.So that, every time you wish to start the server, you can just open up the command prompt and fire up the npm command

Inside the scripts folder, create a new file called runServer.js. So, this script will essentially run the docker command, but before we do so, we must first configure it. Folder structure should resemble something like this.

The very First step is to execute command from the child process of node.js.

// runServer.js
const { exec } = require("child_process");

Next, let's set up the command. We'll use docker -run and detach to run the docker in the background, then assign the port. The most important section of this command is the volume section. What we're doing here is binding the volume, which is essentially a way to store the data of the typesense, which by default is inside the container itself.So, typesense will store the data inside the container's data folder named /data, and it will expose and link this /data folder into the folder we designate on our system, which in this case will be the tmp directory. This approach will assist to keep the data consistent and ensure that it is resilient, and we will always preserve the data so that if the docker container is deleted, stopped, or something similar happens, we will simply keep our data in a secure place. By specifying a -v volume, we are just indicating where we want the data to be stored . After that, we need to specify which image is required to run inside our docker container, so we'll use typesense/typesense:0.22.2 for now, but you can use your own image, and then specify the data directory and the api key, which you can put anything you want and pass the listen port, and finally enable CORS so we don't have any CORS related issues. This is what your code should look like.

// runServer.js
const { exec } = require("child_process");

const command = `docker run -d -p 8108:8108 -v/tmp/typesense-server-data/:/data \ typesense/typesense:0.22.2 --data-dir /data --api-key=animesearch --listen-port 8108 --enable-cors`;

Finally, we must run the command and simply construct an error handler in the event that an error happens while running the commands. As a result, your final "runServer" code should look like this.

// runServer.js
const { exec } = require("child_process");

const command = `docker run -d -p 8108:8108 -v/tmp/typesense-server-data/:/data \ typesense/typesense:0.22.2 --data-dir /data --api-key=animesearch --listen-port 8108 --enable-cors`;

exec(command, (err) => {
  if (!err) console.log("Typesense Server is up and running...✰✨");

  if (err) {
    console.log("Error running server: ", err);
  }
});

So now that we have our runServer.js script ready to go, we can simply update the script tag in our package.json file.Finally, your package.json file should look like this once you've updated the script.

// package.json
{
  "name": "anime-searchapp-reactjs-typesense",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@testing-library/jest-dom": "^5.14.1",
    "@testing-library/react": "^12.0.0",
    "@testing-library/user-event": "^13.2.1",
    "instantsearch.css": "^7.4.5",
    "react": "^17.0.2",
    "react-dom": "^17.0.2",
    "react-instantsearch-dom": "^6.22.0",
    "react-scripts": "5.0.0",
    "styled-components": "^5.3.3",
    "typesense": "^1.1.3",
    "typesense-instantsearch-adapter": "^2.3.0",
    "web-vitals": "^2.1.0"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject",
    "start-server": "node scripts/runServer.js"
  },
  "eslintConfig": {
    "extends": ["react-app", "react-app/jest"]
  },
  "browserslist": {
    "production": [">0.2%", "not dead", "not op_mini all"],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

Finally, start the server by typing npm run start-server, and your docker will start running, then you can proceed to the next step of indexing the data/collection.

The next step is to actually populate or index the data inside the typesense server, so now that we have the server up and running and a bunch of anime data, this is the datset that we want to populate inside the typesense server so that later on in our UI we can basically query that typesense server and do an instant search that will be legit incredibly fast.So to do that let's get started writing the data-importing scripts. We'll start by creating a file called loadData.js inside the scripts folder which we previously created in which we'll initialize the typesense client.

Indexing data on typesense: a step-by-step guide

First step: We need to import the Typesense library to our project.

// loadData.js
const Typesense = require("typesense");

Seecond step: Let's go ahead and make a self-executing function that will run whenever we run the script and make it asynchronous so we can use the async await functionality. Simply create'module.export' and export the self-executing function inside it and make it asynchronous so we can make a script that reads the data and does the collection, manages the typesense server, and indexes the data. So the first thing we need to do is set up the typesense client so that we can connect to the server and begin managing, indexing, and retrieving data. So first create a typesense config variable and pass the nodes properties first. So what it does is it allows you to have multiple nodes for a single server, for example, a client can connect to multiple nodes, and nodes are basically just servers, so this specific nodes is actually an array that holds the actual configuration for each server that you actually want to connect into and that you want the client to access, so currently we only have one server running so we are going to usitlize only one nodes.Next, inside the nodes array, supply the host, typesense port, and protocol it uses, as well as the api key.

// loadData.js
const Typesense = require("typesense");

module.exports = (async () => {
  const TYPESENSE_CONFIG = {
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "animesearch",
  };
})();

Third step: Let's utilize the typesense configuration to make a Typesense client.

// loadData.js
const Typesense = require("typesense");

module.exports = (async () => {
  const TYPESENSE_CONFIG = {
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "animesearch",
  };

  console.log("Config: ", TYPESENSE_CONFIG);

  const typesense = new Typesense.Client(TYPESENSE_CONFIG);
})();

Fourth step: This is a crucial stage because it is here that we provide the schema that will have our data indexed into the actual typesense database, therefore schema is quite critical. It's fairly basic syntax and super simple to work with. Schema is basically where you describe how your data is going to be saved. For our schema, we have title, synopsis, genre, aired, popularity, ranking, score, img url, and links. So in schema you only want to put the fields you wish to index.If you're familiar with nosql databases, especially mongodb, it's heavily inspired by this approach. If you're familiar with the mongoose ORM(Object Relational Model) for example: how it works and how you can retrieve data and collections, it basically works the same way and and typesense has this particular exact feature. So it's basically like a nosql database . It's a little more sophisticated than that, but you may think of it like this to get a general impression of it and understand how the data flows, and how it's all organized together.Begin by giving the schema a name and making sure the number of documents is set to zero. Next, add the fields, which will be an array of objects containing every single field that we want to index and store in our database, so provide the name, type, and facet. So, if you're wondering what facet is, it's a feature that allows you to define categories based on a subset of attributes so that users may narrow down their search results.This is how your schema should look like.

// loadData.js
const Typesense = require("typesense");

module.exports = (async () => {
  const TYPESENSE_CONFIG = {
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "animesearch",
  };

  console.log("Config: ", TYPESENSE_CONFIG);

  const typesense = new Typesense.Client(TYPESENSE_CONFIG);

  const schema = {
    name: "animes",
    num_documents: 0,
    fields: [
      {
        name: "title",
        type: "string",
        facet: false,
      },
      {
        name: "synopsis",
        type: "string",
        facet: false,
      },
      {
        name: "genre",
        type: "auto",
        facet: true,
      },
      {
        name: "genre.lvl0",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl1",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl2",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl3",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl4",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl5",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "aired",
        type: "string",
        facet: true,
      },
      {
        name: "popularity",
        type: "float",
        facet: true,
      },
      {
        name: "ranked",
        type: "float",
        facet: true,
      },
      {
        name: "score",
        type: "string",
        facet: true,
      },
      {
        name: "img_url",
        type: "string",
        facet: true,
      },
      {
        name: "link",
        type: "string",
        facet: true,
      },
    ],
    default_sorting_field: "popularity",
  };
})();

So, if you want to index all of the data in the array of genres, for example, you'll need to store store each level of the array on threir specific/own field.

Let's proceed to our dataset and look at the genres section. As you can see in the figure below, there are four items in that array, so we will make each field level for each of them items.

Fifth step: Let's start by reading the movies from the json files and then importing the dataset. Now is the time to call the Typesense client and establish a connection with a schema within it.

// loadData.js
const Typesense = require("typesense");

module.exports = (async () => {
  const TYPESENSE_CONFIG = {
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "animesearch",
  };

  console.log("Config: ", TYPESENSE_CONFIG);

  const typesense = new Typesense.Client(TYPESENSE_CONFIG);

  const schema = {
    name: "animes",
    num_documents: 0,
    fields: [
      {
        name: "title",
        type: "string",
        facet: false,
      },
      {
        name: "synopsis",
        type: "string",
        facet: false,
      },
      {
        name: "genre",
        type: "auto",
        facet: true,
      },
      {
        name: "genre.lvl0",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl1",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl2",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl3",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl4",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl5",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "aired",
        type: "string",
        facet: true,
      },
      {
        name: "popularity",
        type: "float",
        facet: true,
      },
      {
        name: "ranked",
        type: "float",
        facet: true,
      },
      {
        name: "score",
        type: "string",
        facet: true,
      },
      {
        name: "img_url",
        type: "string",
        facet: true,
      },
      {
        name: "link",
        type: "string",
        facet: true,
      },
    ],
    default_sorting_field: "popularity",
  };

  const animes = require("../dataset/animes.json");

  try {
    const collection = await typesense.collections("animes").retrieve();
    console.log("Found existing collection of animes");
    console.log(JSON.stringify(collection, null, 2));
  } catch (err) {
    console.error(err);
  }
})();

Sixth step: If an error occurs while loading the data(data duplication error), simply add the following snippet of code to the loadData.js file before creating schema becasue it will simply remove the existing data and populates it with the new ones.

if (collection.num_documents !== animes.length) {
  console.log("Collection has diff number of docs than data");
  console.log("Deleting collection");
  await typesense.collections("animes").delete();
}

Seventh step: Creating a collection named animes . In Typesense a Collection is a set of related Documents that functions similarly to a table in a relational database. We give a collection a name and describe the fields that will be indexed when a document is added to the collection when we create it.

Your final code inside loadData.js file should look like this.

// loadData.js
const Typesense = require("typesense");

module.exports = (async () => {
  const TYPESENSE_CONFIG = {
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "animesearch",
  };

  console.log("Config: ", TYPESENSE_CONFIG);

  const typesense = new Typesense.Client(TYPESENSE_CONFIG);

  const schema = {
    name: "animes",
    num_documents: 0,
    fields: [
      {
        name: "title",
        type: "string",
        facet: false,
      },
      {
        name: "synopsis",
        type: "string",
        facet: false,
      },
      {
        name: "genre",
        type: "auto",
        facet: true,
      },
      {
        name: "genre.lvl0",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl1",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl2",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl3",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl4",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "genre.lvl5",
        type: "auto",
        facet: true,
        optional: true,
      },
      {
        name: "aired",
        type: "string",
        facet: true,
      },
      {
        name: "popularity",
        type: "float",
        facet: true,
      },
      {
        name: "ranked",
        type: "float",
        facet: true,
      },
      {
        name: "score",
        type: "string",
        facet: true,
      },
      {
        name: "img_url",
        type: "string",
        facet: true,
      },
      {
        name: "link",
        type: "string",
        facet: true,
      },
    ],
    default_sorting_field: "popularity",
  };

  const animes = require("../dataset/animes.json");

  try {
    const collection = await typesense.collections("animes").retrieve();
    console.log("Found existing collection of animes");
    console.log(JSON.stringify(collection, null, 2));

    if (collection.num_documents !== animes.length) {
      console.log("Collection has diff number of docs than data");
      console.log("Deleting collection");
      await typesense.collections("animes").delete();
    }
  } catch (err) {
    console.error(err);
  }

  console.log("Creating schema...");
  console.log(JSON.stringify(schema, null, 2));

  await typesense.collections().create(schema);

  console.log("Populating collection data...");

  try {
    const returnData = await typesense
      .collections("animes")
      .documents()
      .import(animes);

    console.log("Return data: ", returnData);
  } catch (err) {
    console.error(err);
  }
})();

So now that we have our loadData.js script ready to go, we can simply update the script tag in our package.json file.Finally, your package.json file should look like this once you've updated the script.

// package.json
{
  "name": "anime-searchapp-reactjs-typesense",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@testing-library/jest-dom": "^5.14.1",
    "@testing-library/react": "^12.0.0",
    "@testing-library/user-event": "^13.2.1",
    "instantsearch.css": "^7.4.5",
    "react": "^17.0.2",
    "react-dom": "^17.0.2",
    "react-instantsearch-dom": "^6.22.0",
    "react-scripts": "5.0.0",
    "styled-components": "^5.3.3",
    "typesense": "^1.1.3",
    "typesense-instantsearch-adapter": "^2.3.0",
    "web-vitals": "^2.1.0"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject",
    "start-server": "node scripts/runServer.js",
    "indexer": "node scripts/loadData.js"
  },
  "eslintConfig": {
    "extends": ["react-app", "react-app/jest"]
  },
  "browserslist": {
    "production": [">0.2%", "not dead", "not op_mini all"],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

Finally, start indexing the data in typesense server by typing npm run indexer, and your data will start getting populated inside the typesense server.

Building a UI and retrieving the data

Let's get started by putting together the UI. Our frontend, or UI, will need to connect to the typesense server and do queries. A Typesense server is as standard and flexible as any other HTTP server. You can basically send the HTTP request, which we will be utilizing here as well, and the client will simply submit a normal HTTP request and it will do the search for you. That is basically how it will operate between a client and the actual typesense server.

So, before we perform any UI work or display or render any components, we must first connect to the server and supply the configuration that we accomplished previously.We can now finally adjust the project to use Typesense.We've got our typesense instance up and running in the background.To get React to use the Typesense adapter, open src/app.js file and create a connection first. Inside that, create a TypesenseInstantsearchAdapter object and add server as a key. Inside that, pass the apiKey and nodes, and inside the nodes, specify the host, port, and protocol. Finally, add the additional search parameter to it and pass the query and queryByWeight which you want to display the indexed document/data accordingly.(Remember that these parameters are passed directly to the Typesense search API endpoint. As a result, any parameters supported by the search endpoint can be passed through it).

The following is an example of what your code should look like.

import React, { useState } from "react";
import TypesenseInstantSearchAdapter from "typesense-instantsearch-adapter";

const typesenseInstantsearchAdapter = new TypesenseInstantSearchAdapter({
  server: {
    apiKey: "animesearch",
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
  },
  additionalSearchParameters: {
    queryBy: "titles,synopsis,genre",
    queryByWeights: "4,2,1",
    numTypos: 3,
    typoTokensThreshold: 1,
  },
});

const App = () => {
  return (
    <>
      <div>App</div>
    </>
  );
};

export default App;

Now that we've completed the configuration, let's move on to creating an interface for our application. To do so, first import the InstantSearch component from the react-instantsearch-dom library, and pass indexName and searchClient as props to that component.

// app.js
import React, { useState } from "react";
import TypesenseInstantSearchAdapter from "typesense-instantsearch-adapter";
import { InstantSearch } from "react-instantsearch-dom";

const typesenseInstantsearchAdapter = new TypesenseInstantSearchAdapter({
  server: {
    apiKey: "animesearch",
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
  },
  additionalSearchParameters: {
    queryBy: "titles,synopsis,genre",
    queryByWeights: "4,2,1",
    numTypos: 3,
    typoTokensThreshold: 1,
  },
});

const App = () => {
  return (
    <>
      <InstantSearch
        indexName="animes"
        searchClient={typesenseInstantsearchAdapter.searchClient}
      ></InstantSearch>
    </>
  );
};

export default App;

If you start your " react.js" application, it will appear empty.

Before we dive into integrating searchbox, let's style our application and segregate the search section, as well as add some style to our interface, so simply follow the code below and wrap it inside the InstantSearch component.

const App = () => {
  return (
    <>
      <InstantSearch
        indexName="animes"
        searchClient={typesenseInstantsearchAdapter.searchClient}
      >
        <div className="search-container">
          <aside className="results-section"></aside>
          <main>Search/result section</main>
        </div>
      </InstantSearch>
    </>
  );
};

Now let's add the SearchBox and Hits components from the react-instantsearch-dom library, so we can directly incorporate those components inside our application.

import TypesenseInstantSearchAdapter from "typesense-instantsearch-adapter";
import ResultTemplate from "./components/ResultTemplate";
import "./App.css";
import {
  InstantSearch,
  SearchBox,
  Hits,
  Configure,
  Pagination,
  SortBy,
  Panel,
  RefinementList,
} from "react-instantsearch-dom";

import "instantsearch.css/themes/satellite.css";

const typesenseInstantsearchAdapter = new TypesenseInstantSearchAdapter({
  server: {
    apiKey: "animesearch",
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
  },
  additionalSearchParameters: {
    queryBy: "title,synopsis,genre",
    queryByWeights: "4,2,1",
    numTypos: 3,
    typoTokensThreshold: 1,
  },
});

const App = () => {
  return (
    <>
      <h1 className="super-title">
        👊🏻👨🏻‍🦲Anime search application built using react👺🔪👿
      </h1>
      <InstantSearch
        indexName="animes"
        searchClient={typesenseInstantsearchAdapter.searchClient}
      >
        <Configure hitsPerPage={12} />
        <div className="search-container">
          <aside className="results-section"></aside>
          <main>
            <SearchBox />
            <div className="searchbox-gap"></div>
            <Hits />
          </main>
        </div>
      </InstantSearch>
    </>
  );
};
export default App;

Simply re-run the application after you've fixed it, and your application should now look like this.

At the moment, our data is shown in json format, same like it is in our index. Let's present the data in a more appealing way, therefore let's make a new component folder and inside that folder create another new file called ResultTemplate.js and pass the the hit props. Finally, show the anime title by simply passing {hit.title} and wrapping it in a simple div tag.

// components/ResultTemplate
import React from "react";

const ResultTemplate = ({ hit }) => {
  return (
    <>
      <div>{hit.title}</div>
    </>
  );
};
export default ResultTemplate;

After you've finished creating a component, simply import it into the App.js file and feed it to the Hit component.

<Hits hitComponent={ResultTemplate} />

Simply re-run the application after you've fixed it, and it should now appear like this.

So, let's add some styling to our app to make the results display in a grid format, so head to the app's main page and inspect the element there.

So the concept here with this reactInstanceSearch library is that those elements have predefined classes names, as you can see each element has an ais-Hits,ais-Hits-lists and then you have ais-Hits-items. so we just need to change the styles and because we're using vanilla css approach, we'll use the grid of four columns so let's save the application and simply re-run it.

/* App.css */
.ais-Hits-list {
  display: grid;
  grid-template-columns: repeat(4, minmax(0, 1fr));
  gap: 1rem;
}

Also, don't forget to make some changes to the styles ResultTemplate before re-running the application.

// components/ResultTemplate
import React from "react";

const ResultTemplate = ({ hit }) => {
  return (
    <>
      <div className="anime-container">
        <h3 className="anime-wrapper">{hit.name}</h3>
      </div>
    </>
  );
};
export default ResultTemplate;

.anime-container {
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
  margin-bottom: 2rem;
}

.anime-wrapper {
  border-radius: 1rem;
}

and the application should now look something like this.

It's time to display some images of our anime movies, so we'll use the "img" tag and simply pass{hit.image_url} as an image source, followed by the styles.

// components/ResultTemplate
import React from "react";

const ResultTemplate = ({ hit }) => {
  return (
    <>
      <div className="anime-container">
        <div className="anime-wrapper">
          <img className="anime-image" src={hit.img_url} alt="movie" />
        </div>
        <h3>{hit.name}</h3>
      </div>
    </>
  );
};
export default ResultTemplate;

Styles

/* App.css */
.anime-container {
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
  margin-bottom: 2rem;
}

.anime-wrapper {
  border-radius: 1rem;
}

.anime-image {
  width: 100%;
  height: 150px;
  object-fit: cover;
}

.ais-Hits-list {
  display: grid;
  grid-template-columns: repeat(4, minmax(0, 1fr));
  gap: 1rem;
}

It's time to set up the list of hits that will appear on your page, simply add the Configure component and specify the hitsPerPage option to configure the list of hits that you want to display in your page. Finally, use the Pagination widgets offered by the react-instantsearch-dom library to add pagination. Also, let's try adding facets. To do so, we'll add a widget, which in the react-instanctsearch-dom library is named RefinementList, and then define the attribute we want to take, which in our instance would be genre, and also for labeling add the Panel component as well so Finally, re-run the application. As a result, your completed code should look like this...

import TypesenseInstantSearchAdapter from "typesense-instantsearch-adapter";
import ResultTemplate from "./components/ResultTemplate";
import "./App.css";
import {
  InstantSearch,
  SearchBox,
  Hits,
  Configure,
  Pagination,
  Panel,
  RefinementList,
} from "react-instantsearch-dom";

const typesenseInstantsearchAdapter = new TypesenseInstantSearchAdapter({
  server: {
    apiKey: "animesearch",
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
  },
  additionalSearchParameters: {
    queryBy: "title,synopsis,genre",
    queryByWeights: "4,2,1",
    numTypos: 3,
    typoTokensThreshold: 1,
  },
});

const App = () => {
  return (
    <>
      <h1 className="super-title">
        👊🏻👨🏻‍🦲Anime search application built using react👺🔪👿
      </h1>

      <InstantSearch
        indexName="animes"
        searchClient={typesenseInstantsearchAdapter.searchClient}
      >
        <Configure hitsPerPage={12} />
        <div className="search-container">
          <aside className="results-section">
            <Panel header="Popularity"></Panel>
            <Panel header="Genre">
              <RefinementList
                attribute="genre"
                transformItems={(items) =>
                  items.map((item) => ({
                    ...item,
                    label: item.label.slice(2, -2),
                  }))
                }
                searchable={true}
                showMore={true}
                limit={10}
                showMoreText="Show more"
                showLessText="Show less"
              />
            </Panel>
            <Panel header="Aired">
              <RefinementList attribute="aired" />
            </Panel>
          </aside>
          <main>
            <SearchBox />
            <div className="searchbox-gap"></div>
            <Hits hitComponent={ResultTemplate} />
            <Pagination />
          </main>
        </div>
      </InstantSearch>
    </>
  );
};

export default App;

and the following is how your application should appear:

Finally, using the same procedures as before, we can add sorting capabilities to the application: Add the items with the label default with the value animes to the SortBy widget/component from react-instantsearch-dom, and then create another label called ranked (asc) with the value animes/sort/popularity:asc, and another label called ranked (desc) with the value animes/sort/popularity:desc.

<SortBy
  items={[
    { label: "Default", value: "animes" },
    {
      label: "ranked (asc)",
      value: "animes/sort/popularity:asc",
    },
    {
      label: "ranked (desc)",
      value: "animes/sort/popularity:desc",
    },
  ]}
  defaultRefinement="animes"
/>

Finally, let's update the template for the information we want to display in our app (for example, title,img_url and genres), so this is how your code should appear.

// components/ResultTemplate
import React from "react";

const ResultTemplate = ({ hit }) => {
  return (
    <>
      <div className="anime-container">
        <div className="anime-wrapper">
          <a href={hit.link} target="_blank">
            <img className="anime-image" src={hit.img_url} alt="movie" />
          </a>
        </div>
        <a href={hit.link} target="_blank">
          <h2 className="anime-title">{hit.title}</h2>
        </a>
        <h3 className="anime-genre">Genres: {hit.genre}</h3>
        <p>{hit.synopsis}</p>
      </div>
    </>
  );
};

export default ResultTemplate;

Add a few more styles to the project to make it appear even better, so the application's styles should look like this.

/* App.css */
@import url("https://fonts.googleapis.com/css2?family=Inter:wght@100;200;300;400;500;600&family=Poppins:ital,wght@0,100;0,200;0,300;1,100;1,200&display=swap");
* {
  font-family: "Poppins", sans-serif;
}
.super-title {
  display: flex;
  justify-content: center;
}

.search-container {
  display: flex;
  padding-right: 10px;
}

.results-section {
  height: 100vh;
  padding-left: 5rem;
  padding-right: 1rem;
  padding-top: 5rem;
}

.anime-container {
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
  margin-bottom: 2rem;
}

.anime-wrapper {
  border-radius: 1rem;
}

.anime-image {
  width: 100%;
  height: 150px;
  object-fit: cover;
}

.searchbox-gap {
  padding: 10px;
}

.ais-Hits-list {
  display: grid;
  grid-template-columns: repeat(4, minmax(0, 1fr));
  gap: 1rem;
}

a:link,
a:visited {
  text-align: center;
  text-decoration: none;
  display: inline-block;
}

Finally, use the react instantsearch library's instantsearch.css style and add it directly to the project. Hence, after you've integrated everything, your complete project's source code should look like this.

// App.js
import TypesenseInstantSearchAdapter from "typesense-instantsearch-adapter";
import ResultTemplate from "./components/ResultTemplate";
import "./App.css";
import {
  InstantSearch,
  SearchBox,
  Hits,
  Configure,
  Pagination,
  SortBy,
  Panel,
  RefinementList,
} from "react-instantsearch-dom";

import "instantsearch.css/themes/satellite.css";

const typesenseInstantsearchAdapter = new TypesenseInstantSearchAdapter({
  server: {
    apiKey: "animesearch",
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
  },
  additionalSearchParameters: {
    queryBy: "title,synopsis,genre",
    queryByWeights: "4,2,1",
    numTypos: 3,
    typoTokensThreshold: 1,
  },
});

const App = () => {
  return (
    <>
      <h1 className="super-title">
        👊🏻👨🏻‍🦲Anime search application built using react👺🔪👿
      </h1>

      <InstantSearch
        indexName="animes"
        searchClient={typesenseInstantsearchAdapter.searchClient}
      >
        <Configure hitsPerPage={12} />
        <div className="search-container">
          <aside className="results-section">
            <Panel header="Popularity">
              <SortBy
                items={[
                  { label: "Default", value: "animes" },
                  {
                    label: "ranked (asc)",
                    value: "animes/sort/popularity:asc",
                  },
                  {
                    label: "ranked (desc)",
                    value: "animes/sort/popularity:desc",
                  },
                ]}
                defaultRefinement="animes"
              />
            </Panel>
            <Panel header="Genre">
              <RefinementList
                attribute="genre"
                transformItems={(items) =>
                  items.map((item) => ({
                    ...item,
                    label: item.label.slice(2, -2),
                  }))
                }
                searchable={true}
                showMore={true}
                limit={10}
                showMoreText="Show more"
                showLessText="Show less"
              />
            </Panel>
            <Panel header="Aired">
              <RefinementList attribute="aired" />
            </Panel>
          </aside>
          <main>
            <SearchBox />
            <div className="searchbox-gap"></div>
            <Hits hitComponent={ResultTemplate} />
            <Pagination />
          </main>
        </div>
      </InstantSearch>
    </>
  );
};

export default App;

Let's take a look at the final version of our typesense-integrated Anime search application.

Entire source code of the application can be found here

Join Aviyel’s community to learn more about the open source project, get tips on how to contribute, and join active dev groups. Aviyel is a collaborative platform that assists open source project communities in monetizing and long-term sustainability. To know more visit Aviyel.com and find great blogs and events, just like this one! Sign up now for early access, and don’t forget to follow us on our socials!

Follow @aviyelHQ or sign-up on Aviyel for early access if you are a project maintainer, contributor, or just an Open Source enthusiast.

Join Aviyel's Discord => Aviyel's world

Twitter =>https://twitter.com/AviyelHq

Live Demo => https://supabase-ecommerce.vercel.app

So, in this tutorial, we'll learn how to build this full-stack app with Next.js, the react framework, NextAuth.js, for implementing passwordless and OAuth authentication, Supabase, for persisting app data into a PostgreSQL database and stashing media files and information, and Prisma, for making it simple to read and write data from and to the database from our app.

This article tutorial covers many topics and technical concepts necessary to build a modern full-stack app, even if this app is a simplified version of a more advanced ecommerce site like Amazon. You should be able to use all of the technologies covered in this tutorial, including react, nextjs, prisma, supabase, and others, but most importantly, you should be able to build any full-stack app using those technologies. You'll go at your own speed and intensity, with us guiding you along the way. After completing this guide, the goal of this article is to provide you with the tools and techniques you'll need to build a similar app on your own.To put it another way, this tutorial will not only teach you how to use those technologies in great detail, but it will also provide you with the proper mixture of principles and application to help you grasp all of the key concepts so that you can proudly build your own apps from scratch later part on this article.

Let's start with the react portion and build our application. The first step is to install Node.js if it isn't already on your computer. So, go to the official Node.js website and download the most recent version. Node js is required to use the node package manager, abbreviated as npm. Now launch your preferred code editor and navigate to the folder. For this article tutorial, we'll be using the VScode code editor.

Setting up SupabaseEcommerce project.

There is a Github repository dedicated to this project, which consists of three branches. Clone the SupabaseEcommerce-starter branch to get started.

The Main branch contains the entire final source code of the application, so clone the SupabaseEcommerce-starter branch if you want to follow along with this tutorial.

git clone --branch SupabaseEcommerce-starter https://github.com/pramit-marattha/SupabaseEcommerce.git

After that, head over to the cloned directory and install the dependencies before starting the Next.js development server:

cd SupabaseEcommerce
yarn add all
yarn dev

You can now check if everything is working properly by going to http://localhost:3000 and editing pages/index.js, then viewing the updated result in your browser.For more information on how to use create-next-app, you can review the create-next-app documentation.

It usually only takes a few minutes to get everything set up. So, for this project we will be using yarn to add packages to a project, which will install and configure everything for us so that we can get started right away with an excellent starter template. It's time to start our development server, so head over to that SupabaseEcommerce folder and type yarn add all and then yarn dev and the browser will instantly open our starter template Next.js appplication.

Your application’s folder structure should look something like this.

So you might be curious about the source of the content. Remember that all of our source code is housed in the pages folder, and react/next will inject it into the root div element. Let’s take a look at our pages folder, which contains some javascript files and one API folder.

Before we dive any further lets actually create a landing page for our site.

so before we even begin first you need to install framer-motion library.

Let's dive in and create a beautiful looking UI for our E-commerce application before we start on the backend integration part. Let's start by making a landing page for the app, and then move on to making a product page for it. So, inside the components folder, create a Layout component and add the following code to it. This component is simply a basic layout for our application that includes a navigation bar & menus as well as the functionality to display our application's registration/login modal.

// components/Layout.js
import { Fragment, useState } from "react";
import { useRouter } from "next/router";
import Head from "next/head";
import Link from "next/link";
import Image from "next/image";
import PropTypes from "prop-types";
import AuthModal from "./AuthModal";
import { Menu, Transition } from "@headlessui/react";
import {
  HeartIcon,
  HomeIcon,
  LogoutIcon,
  PlusIcon,
  UserIcon,
  ShoppingCartIcon,
} from "@heroicons/react/outline";
import { ChevronDownIcon } from "@heroicons/react/solid";

const menuItems = [
  {
    label: "List a new home",
    icon: PlusIcon,
    href: "/list",
  },
  {
    label: "My homes",
    icon: HomeIcon,
    href: "/homes",
  },
  {
    label: "Favorites",
    icon: HeartIcon,
    href: "/favorites",
  },
  {
    label: "Logout",
    icon: LogoutIcon,
    onClick: () => null,
  },
];

const Layout = ({ children = null }) => {
  const router = useRouter();

  const [showModal, setShowModal] = useState(false);

  const user = null;
  const isLoadingUser = false;

  const openModal = () => setShowModal(true);
  const closeModal = () => setShowModal(false);

  return (
    <>
      <Head>
        <title>SupaaShop | A new way to shop!</title>
        <meta name="title" content="SupaaShopp" />
        <link rel="icon" href="/favicon.ico" />
      </Head>

      <div className="min-h-screen flex flex-col font-['Poppins'] bg-[linear-gradient(90deg, #161122 21px, transparent 1%) center, linear-gradient(#161122 21px, transparent 1%) center, #a799cc]">
        <header className="h-28 w-full shadow-lg">
          <div className="h-full container mx-auto">
            <div className="h-full px-5 flex justify-between items-center space-x-5">
              <Link href="/">
                <a className="flex items-center space-x-1">
                  <img
                    className="shrink-0 w-24 h-24 text-primary"
                    src="https://user-images.githubusercontent.com/37651620/158058874-6a86646c-c60e-4c39-bc6a-d81974afe635.png"
                    alt="Logo"
                  />
                  <span className="text-2xl font-semibold tracking-wide text-white">
                    <span className="text-3xl text-success">S</span>upabase
                    <span className="text-3xl text-success">E</span>commerce
                  </span>
                </a>
              </Link>
              <div className="flex items-center space-x-4">
                <Link href="/create">
                  <a className="ml-4 px-4 py-5 rounded-md bg-info text-primary hover:bg-primary hover:text-info focus:outline-none focus:ring-4 focus:ring-primaryfocus:ring-opacity-50  font-semibold transition">
                    Register shop !
                  </a>
                </Link>
                {isLoadingUser ? (
                  <div className="h-8 w-[75px] bg-gray-200 animate-pulse rounded-md" />
                ) : user ? (
                  <Menu as="div" className="relative z-50">
                    <Menu.Button className="flex items-center space-x-px group">
                      <div className="shrink-0 flex items-center justify-center rounded-full overflow-hidden relative bg-gray-200 w-9 h-9">
                        {user?.image ? (
                          <Image
                            src={user?.image}
                            alt={user?.name || "Avatar"}
                            layout="fill"
                          />
                        ) : (
                          <UserIcon className="text-gray-400 w-6 h-6" />
                        )}
                      </div>
                      <ChevronDownIcon className="w-5 h-5 shrink-0 text-gray-500 group-hover:text-current" />
                    </Menu.Button>
                    <Transition
                      as={Fragment}
                      enter="transition ease-out duration-100"
                      enterFrom="opacity-0 scale-95"
                      enterTo="opacity-100 scale-100"
                      leave="transition ease-in duration-75"
                      leaveFrom="opacity-100 scale-100"
                      leaveTo="opacity-0 scale-95"
                    >
                      <Menu.Items className="absolute right-0 w-72 overflow-hidden mt-1 divide-y divide-gray-100 origin-top-right bg-white rounded-md shadow-lg ring-1 ring-black ring-opacity-5 focus:outline-none">
                        <div className="flex items-center space-x-2 py-4 px-4 mb-2">
                          <div className="shrink-0 flex items-center justify-center rounded-full overflow-hidden relative bg-gray-200 w-9 h-9">
                            {user?.image ? (
                              <Image
                                src={user?.image}
                                alt={user?.name || "Avatar"}
                                layout="fill"
                              />
                            ) : (
                              <UserIcon className="text-gray-400 w-6 h-6" />
                            )}
                          </div>
                          <div className="flex flex-col truncate">
                            <span>{user?.name}</span>
                            <span className="text-sm text-gray-500">
                              {user?.email}
                            </span>
                          </div>
                        </div>
                        <div className="py-2">
                          {menuItems.map(
                            ({ label, href, onClick, icon: Icon }) => (
                              <div
                                key={label}
                                className="px-2 last:border-t last:pt-2 last:mt-2"
                              >
                                <Menu.Item>
                                  {href ? (
                                    <Link href={href}>
                                      <a className="flex items-center space-x-2 py-2 px-4 rounded-md hover:bg-gray-100">
                                        <Icon className="w-5 h-5 shrink-0 text-gray-500" />
                                        <span>{label}</span>
                                      </a>
                                    </Link>
                                  ) : (
                                    <button
                                      className="w-full flex items-center space-x-2 py-2 px-4 rounded-md hover:bg-gray-100"
                                      onClick={onClick}
                                    >
                                      <Icon className="w-5 h-5 shrink-0 text-gray-500" />
                                      <span>{label}</span>
                                    </button>
                                  )}
                                </Menu.Item>
                              </div>
                            )
                          )}
                        </div>
                      </Menu.Items>
                    </Transition>
                  </Menu>
                ) : (
                  <button
                    type="button"
                    onClick={openModal}
                    className="ml-4 px-4 py-5 rounded-md bg-info hover:bg-primary focus:outline-none focus:ring-4 focus:ring-primary focus:ring-opacity-50 text-primary hover:text-info font-extrabold transition"
                  >
                    Login
                  </button>
                )}
              </div>
            </div>
          </div>
        </header>

        <main className="flex-grow container mx-auto">
          <div className="px-4 py-12">
            {typeof children === "function" ? children(openModal) : children}
          </div>
        </main>

        <AuthModal show={showModal} onClose={closeModal} />
      </div>
    </>
  );
};

Layout.propTypes = {
  children: PropTypes.oneOfType([PropTypes.node, PropTypes.func]),
};

export default Layout;

Let's create a 'Hero' section of our landing page after you've successfully created a layout for the application. To do so, simply paste the following code into that section. So, in this section, we'll add an image on the right, a large text heading, and two buttons on the left.Note that we are styling our project with the absolute power of tailwind css and framer-motion to add some beautiful transition animation to the image .Since we've already created buttons on our starter template, you won't have to worry about creating them from scratch; instead, you can simply import them from the components and use them.

// components/Hero.js
import React from "react";
import PrimaryButton from "@/components/PrimaryButton";
import SecondaryButton from "@/components/SecondaryButton";
import { motion } from "framer-motion";

const Hero = () => {
  return (
    <div className="max-w-6xl mx-auto py-12 flex flex-col md:flex-row space-y-8 md:space-y-0">
      <div className="w-full md:w-1/2 flex flex-col justify-center items-center">
        <div className="max-w-xs lg:max-w-md space-y-10 w-5/6 mx-auto md:w-full text-center md:text-left">
          <h1 className="font-primary font-extrabold text-white text-3xl sm:text-4xl md:text-5xl md:leading-tight">
            Shop <span className="text-success">whenever</span> and{" "}
            <span className="text-success">however</span> you want from,{" "}
            <span className="text-success">wherever</span> you are..{" "}
          </h1>
          <p className="font-secondary text-gray-500 text-base md:text-lg lg:text-xl">
            SuperbaseEcommerce improves and streamlines your shopping
            experience..
          </p>
          <div className="flex space-x-4">
            <PrimaryButton text="Register" link="/" />
            <SecondaryButton text="Let's Shop!" link="/products" />
          </div>
        </div>
      </div>
      <motion.div
        className="w-full md:w-1/2 transform scale-x-125 lg:scale-x-100"
        initial={{ opacity: 0, translateY: 60 }}
        animate={{ opacity: 1, translateY: 0 }}
        transition={{ duration: 0.8, translateY: 0 }}
      >
        <img
          alt="hero-img"
          src="./assets/shop.svg"
          className="mx-auto object-cover shadow rounded-tr-extraLarge rounded-bl-extraLarge w-full h-96 sm:h-112 md:h-120"
        />
      </motion.div>
    </div>
  );
};

export default Hero;

Now, before re-running the server, import this Hero component into the index.js file and wrap it in the Layout component to see the changes you've made.

// index.js
import Layout from "@/components/Layout";
import Hero from "@/components/Hero";

export default function Home() {
  return (
    <Layout>
      <Hero />
    </Layout>
  );
}

This is how your landing page should appear.

After you've finished with the Hero section, go ahead and create a ShopCards component, where we'll simply list the demo features that this application offers and add some images, so your final code for the ShopCards component should look like this.

// components/ShopCards.js
import React, { useState, useEffect, useRef } from "react";
import { motion } from "framer-motion";

const ShopCards = () => {
  const [tab, setTab] = useState(1);

  const tabs = useRef(null);

  const heightFix = () => {
    if (tabs.current.children[tab]) {
      tabs.current.style.height =
        tabs.current.children[tab - 1].offsetHeight + "px";
    }
  };

  useEffect(() => {
    heightFix();
  }, [tab]);
  return (
    <section className="relative">
      <div
        className="absolute inset-0 pointer-events-none pb-26"
        aria-hidden="true"
      ></div>

      <div className="relative max-w-6xl mx-auto px-4 sm:px-6">
        <div className="pt-12 md:pt-20">
          <div className="max-w-3xl mx-auto text-center pb-12 md:pb-16">
            <h1 className="text-3xl mb-4">Features</h1>
            <p className="text-xl text-gray-500">
              List of features that SuperbaseEcommerce provides.
            </p>
          </div>
          <div className="relative max-w-6xl mx-auto px-4 sm:px-6">
            <div className="pt-12 md:pt-20">
              <div className="max-w-3xl mx-auto text-center pb-6 md:pb-16">
                <div className="" data-aos="zoom-y-out" ref={tabs}>
                  <motion.div
                    className="relative w-full h-full"
                    initial={{ opacity: 0, translateY: 60 }}
                    animate={{ opacity: 1, translateY: 0 }}
                    transition={{ duration: 0.8, translateY: 0 }}
                  >
                    <img
                      alt="hero-img"
                      src="./assets/webShop.svg"
                      className="mx-auto object-cover shadow rounded-tr-extraLarge rounded-bl-extraLarge w-full h-96 sm:h-112 md:h-120"
                    />
                  </motion.div>
                </div>
              </div>
            </div>
          </div>

          <div className="max-w-6xl mx-auto py-12 flex flex-col md:flex-row space-y-8 md:space-y-0">
            <div
              className="max-w-xl md:max-w-none md:w-full mx-auto md:col-span-7 lg:col-span-6 md:mt-6 pr-12"
              data-aos="fade-right"
            >
              <div className="md:pr-4 lg:pr-12 xl:pr-16 mb-8">
                <h3 className="h3 mb-3">All of our awesome features</h3>
                <p className="text-xl text-black"></p>
              </div>
              <div className="mb-8 md:mb-0">
                <a
                  className={`flex items-center text-lg p-5 rounded border transition duration-300 ease-in-out mb-3 ${
                    tab !== 1
                      ? "bg-white shadow-md border-success hover:shadow-lg"
                      : "bg-success border-transparent"
                  }`}
                  href="#0"
                  onClick={(e) => {
                    e.preventDefault();
                    setTab(1);
                  }}
                >
                  <div>
                    <div className="font-bold leading-snug tracking-tight mb-1 text-gray-600">
                      Register/Login Feature
                    </div>
                    <div className="text-gray-600">
                      User can login and save their products for later purchase.
                    </div>
                  </div>
                </a>
                <a
                  className={`flex items-center text-lg p-5 rounded border transition duration-300 ease-in-out mb-3 ${
                    tab !== 2
                      ? "bg-white shadow-md border-purple-200 hover:shadow-lg"
                      : "bg-success border-transparent"
                  }`}
                  href="#0"
                  onClick={(e) => {
                    e.preventDefault();
                    setTab(2);
                  }}
                >
                  <div>
                    <div className="font-bold leading-snug tracking-tight mb-1 text-gray-600">
                      Add to cart
                    </div>
                    <div className="text-gray-600">
                      User can add the products/items to their cart
                    </div>
                  </div>
                </a>
                <a
                  className={`flex items-center text-lg p-5 rounded border transition duration-300 ease-in-out mb-3 ${
                    tab !== 3
                      ? "bg-white shadow-md border-purple-200 hover:shadow-lg"
                      : "bg-success border-transparent"
                  }`}
                  href="#0"
                  onClick={(e) => {
                    e.preventDefault();
                    setTab(3);
                  }}
                >
                  <div>
                    <div className="font-bold leading-snug tracking-tight mb-1 text-gray-600">
                      Security
                    </div>
                    <div className="text-gray-600">
                      Hassle free secure login and registration process.
                    </div>
                  </div>
                </a>
                <a
                  className={`flex items-center text-lg p-5 rounded border transition duration-300 ease-in-out mb-3 ${
                    tab !== 4
                      ? "bg-white shadow-md border-purple-200 hover:shadow-lg"
                      : "bg-success border-transparent"
                  }`}
                  href="#0"
                  onClick={(e) => {
                    e.preventDefault();
                    setTab(4);
                  }}
                >
                  <div>
                    <div className="font-bold leading-snug tracking-tight mb-1 text-gray-600">
                      Personalized shops
                    </div>
                    <div className="text-gray-600">
                      User can create/register their very own shop and add their
                      own products.
                    </div>
                  </div>
                </a>
              </div>
            </div>
          </div>
        </div>
      </div>
    </section>
  );
};

export default ShopCards;

Again, before re-running the server, import this ShopCards component into the index.js file and wrap it in the Layout component & below the Hero component to see the changes you've made.

// index.js
import Layout from "@/components/Layout";
import Hero from "@/components/Hero";
import ShopCards from "@/components/ShopCards";

export default function Home() {
  return (
    <Layout>
      <Hero />
      <ShopCards />
    </Layout>
  );
}

For the time being, this is how your landing page should appear.

Finally, let's add a Footer section, so make a Footer component and paste the code below into it.

// components/Footer.js
import Link from "next/link";

const Footer = () => {
  return (
    <footer>
      <div className="max-w-6xl mx-auto px-4 sm:px-6 pt-10">
        <div className="sm:col-span-6 md:col-span-3 lg:col-span-3">
          <section>
            <div className="max-w-6xl mx-auto px-4 sm:px-6">
              <div className="pb-12 md:pb-20">
                <div
                  className="relative bg-success rounded py-10 px-8 md:py-16 md:px-12 shadow-2xl overflow-hidden"
                  data-aos="zoom-y-out"
                >
                  <div
                    className="absolute right-0 bottom-0 pointer-events-none hidden lg:block"
                    aria-hidden="true"
                  ></div>

                  <div className="relative flex flex-col lg:flex-row justify-between items-center">
                    <div className="text-center lg:text-left lg:max-w-xl">
                      <h6 className="text-gray-600 text-3xl font-medium mb-2">
                        Sign-up for the early access!{" "}
                      </h6>
                      <p className="text-gray-100 text-lg mb-6">
                        SuperbaseEcommerce improves and streamlines your
                        shopping experience.. !
                      </p>
                      <form className="w-full lg:w-auto">
                        <div className="flex flex-col sm:flex-row justify-center max-w-xs mx-auto sm:max-w-xl lg:mx-0">
                          <input
                            type="email"
                            className="w-full appearance-none bg-purple-100 border border-gray-700 focus:border-gray-600 rounded-sm px-4 py-3 mb-2 sm:mb-0 sm:mr-2 text-black placeholder-gray-500"
                            placeholder="Enter your email…"
                            aria-label="Enter your email…"
                          />
                          <a
                            className="btn text-white bg-info hover:bg-success shadow"
                            href="#"
                          >
                            Sign-Up!
                          </a>
                        </div>
                      </form>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </section>
        </div>
        <div className="md:flex md:items-center md:justify-between py-4 md:py-8 border-t-2 border-solid">
          <ul className="flex mb-4 md:order-1 md:ml-4 md:mb-0">
            <li>
              <Link
                href="#"
                className="flex justify-center items-center text-blue-400 hover:text-gray-900 bg-blue-100 hover:bg-white-100 rounded-full shadow transition duration-150 ease-in-out"
                aria-label="Twitter"
              >
                <svg
                  className="w-8 h-8 fill-current "
                  viewBox="0 0 32 32"
                  xmlns="http://www.w3.org/2000/svg"
                >
                  <path d="M24 11.5c-.6.3-1.2.4-1.9.5.7-.4 1.2-1 1.4-1.8-.6.4-1.3.6-2.1.8-.6-.6-1.5-1-2.4-1-1.7 0-3.2 1.5-3.2 3.3 0 .3 0 .5.1.7-2.7-.1-5.2-1.4-6.8-3.4-.3.5-.4 1-.4 1.7 0 1.1.6 2.1 1.5 2.7-.5 0-1-.2-1.5-.4 0 1.6 1.1 2.9 2.6 3.2-.3.1-.6.1-.9.1-.2 0-.4 0-.6-.1.4 1.3 1.6 2.3 3.1 2.3-1.1.9-2.5 1.4-4.1 1.4H8c1.5.9 3.2 1.5 5 1.5 6 0 9.3-5 9.3-9.3v-.4c.7-.5 1.3-1.1 1.7-1.8z" />
                </svg>
              </Link>
            </li>
            <li className="ml-4">
              <Link
                href="#"
                className="flex justify-center items-center text-white hover:text-gray-900 bg-black hover:bg-white-100 rounded-full shadow transition duration-150 ease-in-out"
                aria-label="Github"
              >
                <svg
                  className="w-8 h-8 fill-current"
                  viewBox="0 0 32 32"
                  xmlns="http://www.w3.org/2000/svg"
                >
                  <path d="M16 8.2c-4.4 0-8 3.6-8 8 0 3.5 2.3 6.5 5.5 7.6.4.1.5-.2.5-.4V22c-2.2.5-2.7-1-2.7-1-.4-.9-.9-1.2-.9-1.2-.7-.5.1-.5.1-.5.8.1 1.2.8 1.2.8.7 1.3 1.9.9 2.3.7.1-.5.3-.9.5-1.1-1.8-.2-3.6-.9-3.6-4 0-.9.3-1.6.8-2.1-.1-.2-.4-1 .1-2.1 0 0 .7-.2 2.2.8.6-.2 1.3-.3 2-.3s1.4.1 2 .3c1.5-1 2.2-.8 2.2-.8.4 1.1.2 1.9.1 2.1.5.6.8 1.3.8 2.1 0 3.1-1.9 3.7-3.7 3.9.3.4.6.9.6 1.6v2.2c0 .2.1.5.6.4 3.2-1.1 5.5-4.1 5.5-7.6-.1-4.4-3.7-8-8.1-8z" />
                </svg>
              </Link>
            </li>
          </ul>

          <div className="flex-shrink-0 mr-2">
            <Link href="/" className="block" aria-label="SuperbaseEcommerce">
              <img
                className="object-cover h-20 w-full"
                src="https://user-images.githubusercontent.com/37651620/159121520-fe42bbf1-a2af-4baf-bdd8-7efad8523202.png"
                alt="SupabaseEcommerce"
              />
            </Link>
          </div>
        </div>
      </div>
    </footer>
  );
};

export default Footer;

Note: Again, before re-running the server, import this Footer component into the index.js file and wrap it in the Layout component & below the ShopCards component to see the changes you've made.

// index.js
import Layout from "@/components/Layout";
import Hero from "@/components/Hero";
import ShopCards from "@/components/ShopCards";
import Footer from "@/components/Footer";

export default function Home() {
  return (
    <Layout>
      <Hero />
      <ShopCards />
      <Footer />
    </Layout>
  );
}

So, if you re-run the server, this is what your application should look like.

The structure of your component folders should resemble something like this.

Congratulationss!! Now that you've successfully created a landing page for the application, let's move on to the core of the matter: creating the product section of the application.

So, Now let’s look at the _app.js file.

// _app.js
import "../styles/globals.css";
import { Toaster } from "react-hot-toast";

function MyApp({ Component, pageProps }) {
  return (
    <>
      <Component {...pageProps} />
      <Toaster />
    </>
  );
}

export default MyApp;

The App component is used by Next.js to create pages. You can control the page initialization by simply overriding it. It allows you to do amazing things like: Persisting layout across page changes, Keeping state while navigating pages, Custom error handling using componentDidCatch,Inject additional data into pages and Add global styles/CSS are just a few of the great things you can accomplish with it.

In the above \_app.js code the Component parameter represents the active page, when you switch routes, Component will change to the new page. As a result, the page will receive any props you pass to Component. Meanwhile pageProps is an empty object that contains the initial props that were preloaded for your page by one of the data fetching methods.

Now, inside the pages folder, create a new page called products.js and import the Layout and Grid components, then import the data.json file as products and make the following changes to it.

// pages/products.js
import Layout from "@/components/Layout";
import Grid from "@/components/Grid";

import products from "data.json";

export default function Products() {
  return (
    <Layout>
      <div className="mt-8 p-5">
        <Grid products={products} />
      </div>
    </Layout>
  );
}

Database Configurations

Before jumping directly on our application, we'll be utilizing the power of Supabase to create a PostgreSQL database, the Prisma schema to define the app data model, and Next.js to connect those two together. So, let's get started building our database.

Supabase Configurartion

Creating a PostgreSQL database in Supabase is as simple as starting a new project. Head over to supabase.com and Sign-in to your account.

After you've successfully signed in, you should see something similar to this.

Now, select New project button. Fill in your project's required details and again click Create Project button and wait for the new database to load.

After the supabase configured the project, your dashboard should look something similar to this.

Creating a connection URL

Follow the steps outlined below to retrieve your database connection URL after your database has been successfully created. We'll need it to use Prisma in our Next.js app to query and create data.

• Step1 : Head over to the Settings tab(Located at the left side)

• Step2 : Click the Database tab in the sidebar (Located on the left side)

• Step3 : Head over to the bottom of the page to find the Connection string section, then select Nodejs and copy the URL.

Initializing Prisma

Prisma is a next-generation ORM that can be used in Node.js and TypeScript applications to access a database. W eare going to use prisma fo our application because it includes all of the code we need to run our queries. It will save us a lot of time and keep us from having to write a bunch of boilerplate codes.

Installing prisma

Prisma CLI installation

The Prisma command line interface (CLI) is the primary command-line interface for interacting with your Prisma project. It can create new project assets, generate Prisma Client, and analyze existing database structures via introspection to create your application models automatically.

npm i prisma

Initialize prisma

Once you've installed the Prisma CLI, run the following command to get Prisma started in your Next.js application. It will then create a /prisma directory and the schema.prisma file within it inside your particular project folder. so, inside it we will be adding all the configuration for our application.

npx prisma init

// prisma.schema
generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

Note: schema.prisma uses Prisma Schema Language(PSL)

Prisma-client-js, the Prisma JavaScript client, is the configured client represented by the generator block.

generator client {
  provider = "prisma-client-js"
}

Next one is the the provider property of this block represents the type of database we want to use, and the connection url represents how Prisma connects to it.

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

Environment variable

Using environment variables in the schema allows you to keep secrets out of the schema file which in turn improves the portability of the schema by allowing you to use it in different environments. Environment variables is created automatically after we fire the npx prisma init command.

Note: Prisma supports the native connection string format for PostgreSQL, MySQL, SQLite, SQL Server, MongoDB (Preview) and CockroachDB (Preview).

DATABASE_URL="postgresql://test:test@localhost:5432/test?schema=foo"

As you can see, there is an DATABASE_URL variable with a dummy connection URL in this environment variable .env. So, replace this value with the connection string you obtained from Supabase.

DATABASE_URL="postgresql://postgres:[YOUR-PASSWORD]@db.bboujxbwamqvgypibdkh.supabase.co:5432/postgres"

Prisma schemas and models

We can begin working on our application's data models now that database is finally connected to your Next.js. In Prisma, our application models should be defined within the Prisma schema using the Prisma models. These models represent the entities of our application and are defined by the model blocks in the schema.prisma file. Each block contains several fields that represent the data for each entity. So, let's begin by creating the Product model, which will define the data schema for our products properties.

Defining models

Models represent the entities of your application domain. Models are represented by model blocks and define a number of fields. In this data model, Product is the model.

// prisma.schema
generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model Product {
  id          String   @id @default(cuid())
  image       String?
  title       String
  description String
  status      String?
  price       Float
  authenticity        Int?
  returnPolicy        Int?
  warranty       Int?
  createdAt   DateTime @default(now())
  updatedAt   DateTime @updatedAt
}

Each field, as shown in our Product model, has at least a name and its type. To learn more about the Scalar types and Prisma schema refrences visit the following links .

• Data model
• Prisma schema
• Prisma schema reference

Generate Prisma Client

After designning Prisma model, we can begin generating our Prisma Client. We'll need to use Prisma's JavaScript library later in the article to interact with our data from within our Next.js app without having to write all of the SQL queries ourselves. But there's more to it. Prisma Client is, in fact, an auto-generated type-safe API designed specifically for our application which will gives us the JavaScript code we need to run queries on our data.

• Step 1: Installing prisma client

  npm install @prisma/client
  

  

• step2: Generating Prisma client

  npx prisma generate
  

  

The @prisma/client npm package

The @prisma/client npm package consists of two key parts:

• The @prisma/client module itself, which only changes when you re-install the package
• The .prisma/client folder, which is the default location for the unique Prisma Client generated from your schema

@prisma/client/index.d.ts exports .prisma/client

Finally, after you have done that inside your ./node_modules folder, you should now find the generated Prisma Client code.

Note: You need to re-run the prisma generate command after every change that's made to your Prisma schema to update the generated Prisma Client code.

Here is a graphical illustration of the typical workflow for the Prisma Client generation:

Note also that prisma generate is automatically invoked when you're installing the @prisma/client npm package.

The Prisma Client is generated from the Prisma schema and is unique to your project. Each time you change the schema and run prisma generate, the client code changes itself.

Pruning in Node.js package managers has no effect on the .prisma folder.

Creating a table in Supabase

If you look at your database in Supabase, you'll notice there is no table inside it. It's because we haven't yet created the Product table.

The Prisma model we defined in our schema.prisma file has not yet been reflected in our database. As a result, we must manually push changes to our data model to our database.

Pushing the data model

Prisma makes it really very easy to synchonize the schema with our database.So to do that follow the command listed below.

npx prisma db push

This command is only good for prototyping on the schemas locally.

OR,

npx prisma migrate dev

This method (npx prisma migrate dev) will be used in this article because it is very useful in that it allows us to directly sync our Prisma schema with our database while also allowing us to easily track the changes that we make.

So, to begin using Prisma Migrate, enter the following command into the command prompt and after that enter a name for this first migration when prompted.

After you have completed this process successfully, prisma will automatically generate SQL database migration files, and you should be able to see the SQL which should look something like this if you look inside the prisma folder.

-- CreateTable
CREATE TABLE "Product" (
    "id" TEXT NOT NULL,
    "image" TEXT,
    "title" TEXT NOT NULL,
    "description" TEXT NOT NULL,
    "status" TEXT NOT NULL,
    "price" DOUBLE PRECISION NOT NULL,
    "authenticity" INTEGER,
    "returnPolicy" INTEGER,
    "warranty" INTEGER,
    "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
    "updatedAt" TIMESTAMP(3) NOT NULL,

    CONSTRAINT "Product_pkey" PRIMARY KEY ("id")
);
---

Finally, check the Supabase dashboard to see if everything has been successfully synced.

Prisma Studio

Prisma Studio is a visual interface to the data residing inside your database where you can use to quickly visualize and manipulate the data. The cool thing about it is that it runs in entirely on your browser and you don't need to set up any connections because it's already comes with the prisma package. Not only that, from the studio, you can quickly open all of your application's models and interact with them directly via. studio itself.

Note: There is also desktop application available to download

Launching Prisma Studio

Launching the prisma studio is really very easy. Literally all you have to do is run the following command from a Prisma project.

npx prisma studio

Now, open your browser and head over to http://localhost:5555/. You should be able to see the single table that we've created previously if you've followed all of the steps correctly.

Manually adding the records

Lets manually add some records and save the changes that we made.

Finally, lets create a functionality to access that data from within our Next.js app, where we can create new records, update existing ones, and delete old ones.

Interacting with data using Next.js

You should see some demo datas if you look at the Product page of your application.

Now, open the file pages/products.js, file which represents our app's product page.

// pages/products.js
import Layout from "@/components/Layout";
import Grid from "@/components/Grid";

import products from "products.json";

export default function Products() {
  return (
    <Layout>
      <div className="mt-8 p-5">
        <Grid products={products} />
      </div>
    </Layout>
  );
}

As you can see, products data is comming from products.json file.

// products.json
[
  {
    "id": "001",
    "image": "/products/ballpen_300.png",
    "title": "Ball Pen",
    "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
    "authenticity": 100,
    "returnPolicy": 1,
    "status": "new",
    "warranty": 1,
    "price": 50
  },
  {
    "id": "002",
    "image": "/products/actioncamera_300.png",
    "title": "Go-pro cam",
    "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
    "authenticity": 100,
    "returnPolicy": 1,
    "status": "new",
    "warranty": 1,
    "price": 30
  },
  {
    "id": "003",
    "image": "/products/alarmclock_300.png",
    "title": "Alarm Clock",
    "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
    "authenticity": 100,
    "returnPolicy": 1,
    "status": "new",
    "warranty": 1,
    "price": 20
  },
  {
    "id": "004",
    "image": "/products/bangle_600.png",
    "title": "Bangle",
    "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
    "authenticity": 100,
    "returnPolicy": 1,
    "status": "new",
    "warranty": 2,
    "price": 200
  },
  {
    "id": "005",
    "image": "/products/bed_600.png",
    "title": "Large Bed",
    "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
    "authenticity": 100,
    "returnPolicy": 1,
    "status": "out of stock!",
    "warranty": 1,
    "price": 105
  },
  {
    "id": "006",
    "image": "/products/binderclip_600.png",
    "title": "Binder clip",
    "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
    "authenticity": 100,
    "returnPolicy": 2,
    "status": "new",
    "warranty": 1,
    "price": 2
  },
  {
    "id": "007",
    "image": "/products/beyblade_600.png",
    "title": "BeyBlade Burst",
    "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
    "authenticity": 100,
    "returnPolicy": 1,
    "status": "out of stock!",
    "warranty": 1,
    "price": 15
  },
  {
    "id": "008",
    "image": "/products/boxinggloves_600.png",
    "title": "Boxing gloves",
    "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
    "authenticity": 100,
    "returnPolicy": 2,
    "status": "new",
    "warranty": 1,
    "price": 45
  }
]

This data & information is then passed as a prop from the Product component to the Grid component. The Grid component is then in charge of rendering those data as a grid of Card on the screen.

// Products.js
import PropTypes from "prop-types";
import Card from "@/components/Card";
import { ExclamationIcon } from "@heroicons/react/outline";

const Grid = ({ products = [] }) => {
  const isEmpty = products.length === 0;

  return isEmpty ? (
    <p className="text-purple-700 bg-amber-100 px-4 rounded-md py-2 max-w-max inline-flex items-center space-x-1">
      <ExclamationIcon className="shrink-0 w-5 h-5 mt-px" />
      <span>No data to be displayed.</span>
    </p>
  ) : (
    <div className="grid md:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 gap-6">
      {products.map((product) => (
        <Card key={product.id} {...product} onClickFavorite={toggleFavorite} />
      ))}
    </div>
  );
};

Grid.propTypes = {
  products: PropTypes.array,
};

export default Grid;

Now we want to retrieve data from our database, and we'll do so using Server-Side Rendering (SSR). The ability of an application to convert HTML files on the server into a fully rendered HTML page for the client is known as server-side rendering (SSR). The web browser sends a request for information to the server, which responds immediately by sending the client a fully rendered page.

So, in order to use Server Side Rendering(SSR) with Next.js, we must export an asynchronous function getServerSideProps from within the file, which exports the page where we want to render out our data. The data returned by the getServerSideProps function will then be used by Next.js to pre-render our page on each individual request. Let's get started and export this function from our applicartion's Prodcuts page.

// pages/products.js
import Layout from "@/components/Layout";
import Grid from "@/components/Grid";

import products from "products.json";

export async function getServerSideProps() {
  return {
    props: {
      // props for the Home component
    },
  };
}

export default function Products() {
  return (
    <Layout>
      <div className="mt-8 p-5">
        <Grid products={products} />
      </div>
    </Layout>
  );
}

To get the data from supabase, import and instantiate the generated Prisma client.

// pages/products.js
import Layout from "@/components/Layout";
import Grid from "@/components/Grid";
import { PrismaClient } from "@prisma/client";

import products from "products.json";

const prisma = new PrismaClient();

export async function getServerSideProps() {
  return {
    props: {
      // props for the Home component
    },
  };
}

export default function Products() {
  return (
    <Layout>
      <div className="mt-8 p-5">
        <Grid products={products} />
      </div>
    </Layout>
  );
}

Now, Using the findMany query, we can get all of the records in our Product table:

// pages/products.js
import Layout from "@/components/Layout";
import Grid from "@/components/Grid";
import { PrismaClient } from "@prisma/client";

const prisma = new PrismaClient();

export async function getServerSideProps() {
  const products = await prisma.product.findMany();
  return {
    props: {
      products: JSON.parse(JSON.stringify(products)),
    },
  };
}

export default function Products({ products = [] }) {
  return (
    <Layout>
      <div className="mt-8 p-5">
        <Grid products={products} />
      </div>
    </Layout>
  );
}

Simply re-run the application, but if you get an error that looks like the one below, you'll need to regenerate the prisma and then re-run the server.

As you can see, its fixed now

Finally, your application should resemble something like this:

Lets give users the functionality to actually create records from the application itself. So, first step is to actually create.

Create a new records

Head over to the pages/ folder and make a new file called addProduct.js.

// addProducts.js
import Layout from "@/components/Layout";
import ProductList from "@/components/ProductList";

const addProducts = () => {
  const createProduct = () => null;

  return (
    <Layout>
      <div className="max-w-screen-xl mx-auto flex-col">
        <h1 className="text-3xl font-medium text-gray-200 justify-center">
          Add your Products
        </h1>
        <div className="mt-8">
          <ProductList
            buttonText="Add Product"
            redirectPath="/products"
            onSubmit={createProduct}
          />
        </div>
      </div>
    </Layout>
  );
};

export default addProducts;

After that head over to the ProductList component and make the following changes to that component.

//components/ProductList.js
import { useState } from "react";
import { useRouter } from "next/router";
import PropTypes from "prop-types";
import * as Yup from "yup";
import { toast } from "react-hot-toast";
import { Formik, Form } from "formik";
import Input from "@/components/Input";
import AddProductImage from "@/components/AddProductImage";

const ProductSchema = Yup.object().shape({
  title: Yup.string().trim().required(),
  description: Yup.string().trim().required(),
  status: Yup.string().trim().required(),
  price: Yup.number().positive().integer().min(1).required(),
  authenticity: Yup.number().positive().integer().min(1).required(),
  returnPolicy: Yup.number().positive().integer().min(1).required(),
  warranty: Yup.number().positive().integer().min(1).required(),
});

const ProductList = ({
  initialValues = null,
  redirectPath = "",
  buttonText = "Submit",
  onSubmit = () => null,
}) => {
  const router = useRouter();

  const [disabled, setDisabled] = useState(false);
  const [imageUrl, setImageUrl] = useState(initialValues?.image ?? "");

  const upload = async (image) => {
    // TODO: Upload image to remote storage
  };

  const handleOnSubmit = async (values = null) => {
    let toastId;
    try {
      setDisabled(true);
      toastId = toast.loading("Submitting...");
      // Submit data
      if (typeof onSubmit === "function") {
        await onSubmit({ ...values, image: imageUrl });
      }
      toast.success("Successfully submitted", { id: toastId });
      // Redirect user
      if (redirectPath) {
        router.push(redirectPath);
      }
    } catch (e) {
      toast.error("Unable to submit", { id: toastId });
      setDisabled(false);
    }
  };

  const { image, ...initialFormValues } = initialValues ?? {
    image: "",
    title: "",
    description: "",
    status: "",
    price: 0,
    authenticity: 1,
    returnPolicy: 1,
    warranty: 1,
  };

  return (
    <div>
      <Formik
        initialValues={initialFormValues}
        validationSchema={ProductSchema}
        validateOnBlur={false}
        onSubmit={handleOnSubmit}
      >
        {({ isSubmitting, isValid }) => (
          <Form className="space-y-6">
            <div className="space-y-6">
              <Input
                name="title"
                type="text"
                label="Title"
                placeholder="Entire your product name..."
                disabled={disabled}
              />

              <Input
                name="description"
                type="textarea"
                label="Description"
                placeholder="Enter your product description...."
                disabled={disabled}
                rows={3}
              />

              <Input
                name="status"
                type="text"
                label="Status(new/out-of-stock/used)"
                placeholder="Enter your product status...."
                disabled={disabled}
              />

              <Input
                name="price"
                type="number"
                min="0"
                label="Price of the product..."
                placeholder="100"
                disabled={disabled}
              />

              <div className="justify-center">
                <Input
                  name="authenticity"
                  type="number"
                  min="0"
                  label="authenticity(%)"
                  placeholder="2"
                  disabled={disabled}
                />
                <Input
                  name="returnPolicy"
                  type="number"
                  min="0"
                  label="returnPolicy(? years)"
                  placeholder="1"
                  disabled={disabled}
                />
                <Input
                  name="warranty"
                  type="number"
                  min="0"
                  label="warranty(? years)"
                  placeholder="1"
                  disabled={disabled}
                />
              </div>
            </div>

            <div className="flex justify-center">
              <button
                type="submit"
                disabled={disabled || !isValid}
                className="bg-success text-white py-2 px-6 rounded-md focus:outline-none focus:ring-4 focus:ring-teal-600 focus:ring-opacity-50 hover:bg-teal-500 transition disabled:opacity-50 disabled:cursor-not-allowed disabled:hover:bg-teal-600"
              >
                {isSubmitting ? "Submitting..." : buttonText}
              </button>
            </div>
          </Form>
        )}
      </Formik>
      <div className="mb-6 max-w-full">
        <AddProductImage
          initialImage={{ src: image, alt: initialFormValues.title }}
          onChangePicture={upload}
        />
      </div>
    </div>
  );
};

ProductList.propTypes = {
  initialValues: PropTypes.shape({
    image: PropTypes.string,
    title: PropTypes.string,
    description: PropTypes.string,
    status: PropTypes.string,
    price: PropTypes.number,
    authenticity: PropTypes.number,
    returnPolicy: PropTypes.number,
    warranty: PropTypes.number,
  }),
  redirectPath: PropTypes.string,
  buttonText: PropTypes.string,
  onSubmit: PropTypes.func,
};

export default ProductList;

After that, go to the AddProductImage file inside the component folder and copy the following code.

// AddProductImage.js
import { useState, useRef } from "react";
import PropTypes from "prop-types";
import Image from "next/image";
import toast from "react-hot-toast";
import classNames from "classnames";
import { CloudUploadIcon } from "@heroicons/react/outline";

const AddProductImage = ({
  label = "Image",
  initialImage = null,
  objectFit = "cover",
  accept = ".png, .jpg, .jpeg, .gif .jiff",
  sizeLimit = 10 * 1024 * 1024,
  onChangePicture = () => null,
}) => {
  const pictureRef = useRef();
  const [image, setImage] = useState(initialImage ?? null);
  const [updatingPicture, setUpdatingPicture] = useState(false);
  const [pictureError, setPictureError] = useState(null);

  const handleOnChangePicture = (e) => {
    const file = e.target.files[0];
    const reader = new FileReader();
    const fileName = file?.name?.split(".")?.[0] ?? "New file";

    reader.addEventListener(
      "load",
      async function () {
        try {
          setImage({ src: reader.result, alt: fileName });
          if (typeof onChangePicture === "function") {
            await onChangePicture(reader.result);
          }
        } catch (err) {
          toast.error("Unable to update image");
        } finally {
          setUpdatingPicture(false);
        }
      },
      false
    );

    if (file) {
      if (file.size <= sizeLimit) {
        setUpdatingPicture(true);
        setPictureError("");
        reader.readAsDataURL(file);
      } else {
        setPictureError("File size is exceeding 10MB.");
      }
    }
  };

  const handleOnClickPicture = () => {
    if (pictureRef.current) {
      pictureRef.current.click();
    }
  };

  return (
    <div className="flex flex-col space-y-2">
      <label className="text-gray-200 ">{label}</label>

      <button
        disabled={updatingPicture}
        onClick={handleOnClickPicture}
        className={classNames(
          "relative aspect-video overflow-hidden rounded-md disabled:opacity-50 disabled:cursor-not-allowed transition group focus:outline-none",
          image?.src
            ? "hover:opacity-50 disabled:hover:opacity-100"
            : "border-2 border-dotted hover:border-gray-400 focus:border-gray-400 disabled:hover:border-gray-200"
        )}
      >
        {image?.src ? (
          <Image
            src={image.src}
            alt={image?.alt ?? ""}
            layout="fill"
            objectFit={objectFit}
          />
        ) : null}

        <div className="flex items-center justify-center">
          {!image?.src ? (
            <div className="flex flex-col items-center space-y-2">
              <div className="shrink-0 rounded-full p-2 bg-gray-200 group-hover:scale-110 group-focus:scale-110 transition">
                <CloudUploadIcon className="w-4 h-4 text-gray-500 transition" />
              </div>
              <span className="text-xs font-semibold text-gray-500 transition">
                {updatingPicture
                  ? "Image Uploading..."
                  : "Upload product Image"}
              </span>
            </div>
          ) : null}
          <input
            ref={pictureRef}
            type="file"
            accept={accept}
            onChange={handleOnChangePicture}
            className="hidden"
          />
        </div>
      </button>

      {pictureError ? (
        <span className="text-red-600 text-sm">{pictureError}</span>
      ) : null}
    </div>
  );
};

AddProductImage.propTypes = {
  label: PropTypes.string,
  initialImage: PropTypes.shape({
    src: PropTypes.string,
    alt: PropTypes.string,
  }),
  objectFit: PropTypes.string,
  accept: PropTypes.string,
  sizeLimit: PropTypes.number,
  onChangePicture: PropTypes.func,
};

export default AddProductImage;

This addProduct component renders the entire page's layout, which consist of a form from where you can add the product details and informations.

API endpoint

Let's actually create a API endpoint that will actually create a new record on our database via addProduct function.

const createProduct = () => null;

But first, within our Next.js application project, let's create an API endpoint to handle our POST request for creating new records. Next.js provides a file based API routing so any file in the pages/api folder is mapped to /api/* and treated as an API endpoint rather than a page. They're only server-side bundles, so they won't add to the size of your client-side bundle. So, create a file name called products.js inside the pages/api folder and inside it create a request handler fucntion like shown below.

export default async function handler(req, res) {}

Handling POST request for products

Before we go any further, use req.method to check the HTTP method of the request inside that request handler function. After that, return a 405 status code to the client becasue we are not handlling any kind of HTTP method.

// pages/api/products.js
export default async function handler(req, res) {
  if (req.method === "POST") {
    // TODO
  } else {
    res.setHeader("Allow", ["POST"]);
    res
      .status(405)
      .json({ message: `HTTP method ${req.method} is not supported.` });
  }
}

Adding new records with Prisma Client

Now, lets use Prisma Client to create a new Product record in the database using the data from the current HTTP request.

// pages/api/products.js
export default async function handler(req, res) {
  if (req.method === "POST") {
    const {
      image,
      title,
      description,
      status,
      price,
      authenticity,
      returnPolicy,
      warranty,
    } = req.body;
  } else {
    res.setHeader("Allow", ["POST"]);
    res
      .status(405)
      .json({ message: `HTTP method ${req.method} is not supported.` });
  }
}

After that, lets actually initialize Prisma and call the create function that prisma provides.

import { PrismaClient } from "@prisma/client";

const prisma = new PrismaClient();

export default async function handler(req, res) {
  if (req.method === "POST") {
    const {
      image,
      title,
      description,
      status,
      price,
      authenticity,
      returnPolicy,
      warranty,
    } = req.body;

    const home = await prisma.product.create({
      data: {
        image,
        title,
        description,
        status,
        price,
        authenticity,
        returnPolicy,
        warranty,
      },
    });
  } else {
    res.setHeader("Allow", ["POST"]);
    res
      .status(405)
      .json({ message: `HTTP method ${req.method} is not supported.` });
  }
}

Finally lets add some try catch block to Handle the error.

// pages/api/products.js
import { PrismaClient } from "@prisma/client";
const prisma = new PrismaClient();

export default async function handler(req, res) {
  if (req.method === "POST") {
    try {
      const {
        image,
        title,
        description,
        status,
        price,
        authenticity,
        returnPolicy,
        warranty,
      } = req.body;

      const product = await prisma.product.create({
        data: {
          image,
          title,
          description,
          status,
          price,
          authenticity,
          returnPolicy,
          warranty,
        },
      });
      res.status(200).json(product);
    } catch (e) {
      res.status(500).json({ message: "Something went wrong" });
    }
  } else {
    res.setHeader("Allow", ["POST"]);
    res
      .status(405)
      .json({ message: `HTTP method ${req.method} is not supported.` });
  }
}

Now that we've created our API, let's call the API endpoint. To do so, open the addProduct.js file in the pages folder and make the following changes to the code, but first, we'll need to install the axios package, so do that first.

npm i axios

OR

yarn add axios

//pages/addProducts.js
import Layout from "@/components/Layout";
import ProductList from "@/components/ProductList";

const addProducts = () => {
  const createProduct = () => (data) => axios.post("/api/products", data);

  return (
    <Layout>
      <div className="max-w-screen-xl mx-auto flex-col">
        <h1 className="text-3xl font-medium text-gray-200 justify-center">
          Add your Products
        </h1>
        <div className="mt-8">
          <ProductList
            buttonText="Add Product"
            redirectPath="/products"
            onSubmit={createProduct}
          />
        </div>
      </div>
    </Layout>
  );
};

export default addProducts;

Now lets re-run the server again.

After that head over to your browser and go to the http://localhost:3000/addProducts route and fill out all the product information and Submit it.

It will automatically redirect you to the /products page and you should be able to see the product that you just added.

Pre-rendering the pages

We've used the getServerSideProps function to pre-render the product of our app using Server-Side Rendering(SSR). Next.js, on the other hand, comes with a built-in pre-rendering method called Static Generation (SSG).

When a page uses Static Generation, the HTML for that page is generated during the build process. That means that when you run next build in production, the page HTML is generated. Each request will then be served with the same HTML. A CDN can cache it. You can statically generate pages with or without data using Next.js.

We can use different pre-rendering techniques on our applications when we use a framework like Next.js. For something more simple and non-dynamic, we can use static site generation(SSG). For dynamic content and more complex pages, we can use server-side rendering(SSR) .

Dynamic Routing with SSG

We can still statically generate pages with SSG after fetching some external data during the build process, even if SSG generates HTML at build time. learn more about static generation and dynamic routing.

Let's get data at build time by exporting an async function called getStaticProps from the pages we want to statically generate.

For Example,

// posts will be populated at build time by getStaticProps()
function Blog({ posts }) {
  return (
    <ul>
      {posts.map((post) => (
        <li>{post.title}</li>
      ))}
    </ul>
  );
}
// This function gets called at build time on server-side.
// It won't be called on client-side, so you can even do
// direct database queries.
export async function getStaticProps() {
  // Call an external API endpoint to get posts.
  // You can use any data fetching library
  const res = await fetch("https://.../posts");
  const posts = await res.json();
  // By returning { props: { posts } }, the Blog component
  // will receive `posts` as a prop at build time
  return {
    props: {
      posts,
    },
  };
}
export default Blog;

Let's put Static Generation(SSG) to work in our application. The pages that render each individual Product listing are the ones that we'll statically generate at the build time. However, because product listings are generated through the users, we could end up with massive amount of pages. As a result, we won't be able to define those routes using predefined paths. Otherwise, we'll end up with a slew of useless files cluttering up our project.

We can easily create dynamic routes in Next.js. We just need to add brackets to a page's filename, [id].js, to create a dynamic route. However, in our project, we will place that in the Products folder. As a result, any route's ids will be matched with their specific id value, and the id value will be available inside the React component that renders the associated page.

Now, go to the pages folder and make a new folder called products, then make a new file called [id].js inside it.

And finally paste the following code inside that file.

// pages/products/[id].jsx
import Image from "next/image";
import Layout from "@/components/Layout";

const ListedProducts = (product = null) => {
  return (
    <Layout>
      <div className="max-w-screen-lg mx-auto">
        <div className="mt-6 relative aspect-video bg-gray-400 rounded-lg shadow-md overflow-hidden">
          {product?.image ? (
            <Image
              src={product.image}
              alt={product.title}
              layout="fill"
              objectFit="cover"
            />
          ) : null}
        </div>
        <div className="flex flex-col sm:flex-row sm:justify-between sm:space-x-4 space-y-4 pt-10">
          <div>
            <h1 className="text-2xl font-semibold truncate">
              {product?.title ?? ""}
            </h1>
            <ol className="inline-flex items-center space-x-1 text-info">
              <li>
                <span aria-hidden="true"> ( </span>
                <span>{product?.status ?? 0} product</span>
                <span aria-hidden="true"> ) </span>
                <span aria-hidden="true"> - </span>
              </li>
              <li>
                <span aria-hidden="true"> ( </span>
                <span>{product?.authenticity ?? 0}% Authentic</span>
                <span aria-hidden="true"> ) </span>
                <span aria-hidden="true"> - </span>
              </li>
              <li>
                <span aria-hidden="true"> ( </span>
                <span>{product?.returnPolicy ?? 0} year return policy</span>
                <span aria-hidden="true"> ) </span>
                <span aria-hidden="true"> - </span>
              </li>
              <li>
                <span aria-hidden="true"> ( </span>
                <span>{product?.warranty ?? 0} year warranty</span>
                <span aria-hidden="true"> ) </span>
              </li>
            </ol>
            <p className="mt-8 text-lg">{product?.description ?? ""}</p>
          </div>
        </div>
      </div>
    </Layout>
  );
};

export default ListedProducts;