How To Send Data From Your PHP CodeBase to Amazon Kinesis

In this tutorial, we will explore how to send data from your PHP codebase to Amazon Kinesis, a real-time data streaming service. We will create a Kinesis stream and then show you how to send a single record to the stream. By the end of this article, you will be able to send data from your PHP codebase to Amazon Kinesis.

Understanding Amazon Kinesis is essential for this process, so let's start by discussing what it is and the benefits it offers.

Understanding Amazon Kinesis

What is Amazon Kinesis?

Amazon Kinesis is a fully managed, scalable, real-time data streaming service provided by Amazon Web Services (AWS). Amazon Kinesis enables you to ingest, process, and analyze streaming data in real time. It is capable of handling large amounts of data, making it suitable for applications such as real-time analytics, log monitoring, and IoT data processing.

Amazon Kinesis is a suite of services designed to handle real-time streaming data on the AWS platform. Each service within the Kinesis family is built for a specific kind of streaming use case. Here's a rundown:

1. Amazon Kinesis Data Streams (KDS):

  • Purpose: This is the foundational service that captures, processes, and stores real-time data. It allows you to build custom applications that process or analyze streaming data.
  • Typical Use Cases: Real-time analytics, logging, monitoring, etc.

2. Amazon Kinesis Data Firehose:

  • Purpose: This service is designed to load streaming data into other AWS services. It's a way to easily capture, transform, and load data streams into data stores without having to write custom code.
  • Destination Services: Amazon S3 (Amazon Simple Storage Service), Amazon Redshift, Amazon Elasticsearch Service, and Splunk.
  • Typical Use Cases: ETL jobs (Extract, Transform, Load), data lakes, log monitoring, etc.
  • Note: With Firehose, you don't manage the underlying stream (like you would with KDS). Instead, you just configure the source and destination, and Firehose handles the rest.

3. Amazon Kinesis Video Streams:

  • Purpose: This service is built specifically for streaming video. It captures, processes, and stores video for analytics and machine learning.
  • Features: It supports various SDKs and devices for video ingestion and can work with other AWS services for video analysis, like Amazon Rekognition Video.
  • Typical Use Cases: Surveillance systems, user-generated content platforms, industrial video monitoring, etc.

4. Amazon Kinesis Data Analytics:

  • Purpose: Allows you to process and analyze streaming data using SQL or Java (via Apache Flink).
  • Integration: Works with both Kinesis Data Streams and Kinesis Data Firehose Stream.
  • Typical Use Cases: Real-time dashboards, metrics generation, anomaly detection, etc.

When considering a streaming solution with Kinesis, it's essential to understand the specific requirements of your use case to select the right service(s). In this article, we will show an example for Data Streams.

Benefits of Using Amazon Kinesis

There are several benefits to using Amazon Kinesis:

Real-time Data Processing:

One of the key advantages of using Amazon Kinesis is its ability to process data in real time. This means that as soon as the data is ingested into the system, it can be immediately processed and analyzed. This allows businesses to gain valuable insights and take immediate action based on the incoming data. For example, a retail company can use Amazon Kinesis to process real-time sales data and adjust pricing or inventory levels accordingly. This real-time processing capability is crucial for applications that require quick decision-making and responsiveness.

Scalability:

Another major benefit of Amazon Kinesis is its scalability. The service is designed to handle any amount of streaming data, from small to large volumes. Whether you have a few kilobytes of data per hour or terabytes of data per hour, Amazon Kinesis can handle it. This scalability makes it suitable for a wide range of use cases, from small-scale applications to enterprise-level systems. Additionally, Amazon Kinesis automatically scales up or down based on the incoming data volume, ensuring that you have the resources you need to process and analyze your data effectively.

Durability and Fault Tolerance:

When it comes to handling data, durability and fault tolerance are critical. Amazon Kinesis provides automatic replication and fault tolerance mechanisms to ensure that your data is safe and available at all times. The service replicates your data across multiple availability zones within a region, protecting against data loss in the event of a failure. Additionally, Amazon Kinesis monitors the health of its components and automatically replaces any failed components to maintain high availability. This durability and fault tolerance feature gives you peace of mind, knowing that your data is protected and accessible even in the face of unexpected events.

Integration with AWS Services:

Amazon Kinesis seamlessly integrates with other AWS services, allowing you to leverage the power of the AWS ecosystem for further analysis and storage of your data. For example, you can easily store your streaming data in Amazon S3 for long-term storage and analysis. You can also use Amazon Redshift, a fully managed data warehouse service, to perform complex analytics on your streaming data. Furthermore, Amazon Kinesis integrates with Amazon Elasticsearch, a fully managed search and analytics engine, enabling you to perform real-time analysis and visualization of your data. This integration with AWS services provides you with a comprehensive and powerful platform for managing and analyzing your streaming data.

In conclusion, Amazon Kinesis is a versatile and powerful service that allows you to collect, process, and analyze large amounts of streaming data in real time. With its real-time data processing capabilities, scalability, durability, fault tolerance, and seamless integration with other AWS services, Amazon Kinesis provides a robust solution for a wide range of use cases, from real-time analytics to IoT data processing.

Setting up Amazon Kinesis

In this section we’ll cover the basic concepts of Amazon Kinesis, the data types it supports, how to set it up, and how to authenticate when using the AWS SDK:

The basic concepts of Amazon Kinesis

In order to understand this tutorial, it is recommended that you get yourself familiar with the basic concepts and keywords used in Kinesis. We picked these top concepts/keywords you should know:

  • Shard: A shard is a base throughput unit in a Kinesis stream. Each shard can handle up to 5 transactions per second for writes and 2 MB/sec data read.
  • Stream: Represents a group of shards. Each stream can be thought of as a continuously updated data source or log file.
  • Partition Key: Used to segregate and route records to different shards of a stream. It's essential for evenly distributing data across shards.
  • Sequence Number: Each record in the shard is assigned a unique sequence number, ensuring order within the shard.

Creating an Amazon Kinesis account

If you haven’t already, create an Amazon Kinesis account. As it is part of Amazon Web Services, you will need to use the AWS console to set up your Amazon Kinesis account as follows:

  • Sign in to the AWS Management Console. If you don’t have an AWS account, you will need to create one.
  • Navigate to the Amazon Kinesis Console.
  • Choose ‘Create Data Stream’ and provide a name for your stream and specify the number of shards you need. You may skip this step if you want to create a stream programmatically, we will cover that guide later in this article.

Data types supported by Amazon Kinesis

Amazon Kinesis supports various data types including JSON data, blob, binary, XML, and CSV.

How to authenticate when using the AWS SDK

To authenticate when using the AWS SDK, you will need to provide your AWS credentials. You can do this by setting the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables or by passing them to the SDK as parameters.

You can also use an IAM role to authenticate. To do this, you will need to provide the role's ARN to the SDK.

Integrating your PHP codebase with Amazon Kinesis

Kinesis provides APIs to interact with Kinesis services. It also provides PHP SDK (`aws/aws-sdk-php`) which makes it more developer-friendly to interact with Amazon Kinesis services. In this guide, we will use the AWS SDK for PHP to integrate PHP with Kinesis.

Configuring AWS PHP SDK

First, you need to install the SDK as a dependency. Here’s how you can install using Composer:

SH
composer require aws/aws-sdk-php

Now that the SDK is installed, you’ll need to import the SDK in your code. Here’s how you can do it using Composer:

PHP
require '/path/to/vendor/ autoload.php';


require 'vendor/autoload.php';


use Aws\Kinesis\KinesisClient;


use Aws\Exception\AwsException;

Writing the PHP code to send data

By writing code to send data from PHP to Kinesis, your PHP app starts acting as a data producer for the stream. In this section, we will learn how to do that using AWS PHP SDK.

If not already present, first create a data stream using `KinesisClient`’s `createStream` function. And then you can use `KinesisClient`’s putRecord` function to save data to that stream.

Check out the following example that creates a Kinesis data stream and sends data to that Kinesis stream:

PHP
<?php


require 'vendor/autoload.php';








use Aws\Kinesis\KinesisClient;


use Aws\Exception\AwsException;








// AWS Configuration


$awsProfile = 'YOUR_AWS_PROFILE'; // replace with your AWS profile name


$awsVersion = 'latest'; // usually 'latest' but can be a specific version if needed


$awsRegion = 'us-east-2'; // replace with the desired AWS region








// Initialize Kinesis Client


$kinesisClient = new KinesisClient([


'profile' => $awsProfile,


'version' => $awsVersion,


'region' => $awsRegion


]);








// Create a data stream


$streamName = "my_stream_name";


$shardCount = 2;








try {


$result = $kinesisClient->createStream([


'ShardCount' => $shardCount,


'StreamName' => $streamName


]);


var_dump($result);


} catch (AwsException $e) {


echo "Error creating the stream: " . $e->getMessage() . "\n";


}








// Send data to an existing data stream


$dataPayload = '{"ticker_symbol":"QXZ", "sector":"HEALTHCARE", "change":-0.05, "price":84.51}'; // The data payload you want to send


$partitionKey = "QXZ"; // Using the ticker symbol as the partition key for this example








try {


$result = $kinesisClient->putRecord([


'Data' => $dataPayload,


'StreamName' => $streamName,


'PartitionKey' => $partitionKey


]);






echo "<p>ShardID = " . $result["ShardId"] . "</p>";


var_dump($result);


} catch (AwsException $e) {


echo "Error sending data to the stream: " . $e->getMessage() . "\n";


}


?>

Let’s understand the options used in `putRecord` method

  • `Data`: The data blob to be added to the record, which is Base64-encoded when the blob is serialized. This data could be any kind of data; for instance, it could be a serialized JSON object, raw bytes from a file, etc.
  • `StreamName`: The name of the Kinesis data stream where the record is to be added.
  • `PartitionKey`: It determines the shard in which the data record will be placed. All data records with the same partition key map to the same shard. Properly understanding and designing partition keys is critical for evenly distributing data across shards and achieving maximum throughput.

There are more options that you can use to meet the requirements for your use case, we won’t discuss them here to keep this example simple. Make sure to read the developer guides provided in the AWS PHP SDK documentation and Kinesis API references to learn more. You may use Amazon Cloudwatch to monitor and track Cloudwatch metrics or alert any issues in this integration.

Testing the Integration

After writing the integration code, it's important to thoroughly test it to ensure data is being successfully sent to Amazon Kinesis. You can validate the integration by monitoring the Kinesis stream and verifying the incoming data. You may use `KinesissClient`’s `describeStream` method to get information about your data stream programmatically.

Conclusion

By following these steps, you can successfully send data from your PHP codebase to Amazon Kinesis. This powerful combination allows you to leverage real-time data processing and analysis capabilities provided by AWS. Start implementing this integration today and unlock the potential of your data!

Don't want to go through the pain of direct integration? RudderStack's PHP SDK makes it easy to send data from your PHP app to Amazon Kinesis.