Post

Stress analysis on Social media

Resources Code: https://github.com/chiranthans23/stress-analysis-social-media Train and test data: dreaddit-train.csv, dreaddit-test.csv EDA report: report.html Introduction This is a work on Dreaddit dataset taken from one of the latest datasets on Kaggle which is created using data from 5 different Reddit communities. This is a slightly different problem as the length of the posts is larger than the usual tweets. Additionally, the data available for training is also not quite huge. First look This work doesn’t have an extensive EDA but just looks at the dataset - columns, amount of data available, missing values.

AWS DynamoDB

AWS DynamoDB Overview Fully managed, highly available and has 3 AZ Integrated with IAM Max size of an item is 400KB Primary key Partition key Partition key+ Sort key RCU and WCU throughput RCU- Read capacity units- throughput for reads WCU- Write capacity units- throughput for writes Option to setup auto-scaling to meet the demands Usig burst credit throughput can be exceeded If burst credit is empty, ProvisionedThroughputException is thrown

AWS SNS and SQS

AWS SNS and SQS Overview AWS service that helps in decoupling the applications and still communicate Unlimited throughput, message stays for 4-14 days in the queue Has a producer and a consumer SQS with ASG Can setup CloudWatch alarm to check on the number of messages in the queue and based on that scale up or down, the number of consumers Create a Queue Management Console-> SQS-> Create Queue We can add messages and when polled, messages will appear in the queue Deleting the message signals that it’s been read Message visibility timeout Time given for a consumer to process a message.

AWS ECS

Amazon ECS Overview ECS clusters are logical grouping of EC2 instances EC2 instances run the ECS agent(Docker container) ECS agents registers the instance to ECS cluster EC2 instances run a special AMI made specifically for ECS Creating a ECS cluster Management console-> ECS Create cluster- providing details like the instance size, VPC and subnet details, security group details and so on We can see the ECS details, auto scaling groups, launch configuration and so on If we had enabled ssh to ec2 instance, we can ssh and see in the docker logs that ec2 instance will be registering itself to ecs cluster that we created ECS task definitions It is metadata in JSON form that tells ECS how to run a Docker container It contains following information: Image name Port binding of container and host Memory and CPU required Env variables Networking information etc Creating an ECS task definition Go to ECS -> task definition -> create task definition Reserve memory and CPU size Provide the image+tag name Add container Set the hard, soft limit for the memory Port mapping , volume mapping, etc We can also see the task definition in JSON form as well Creating an ECS service It helps in defining how the tasks should be run and number of tasks should run ECS clusters-> cluster-> create service Choose the task, cluster, etc and create the service This should start running the task and can be visualised through docker logs in the host We can also scale the number of tasks to be run( if the port is reserved for a task, we need to scale instance number in auto scaling group too) ECS service with LB We can have multiple instances running tasks and a load balancer can be used to stream the traffic among them Note: A service can’t be edited to add load balancer, so a new service should be created with a load balancer attached to it Create a load balancer with appropriate vpc, subnets, security group and set the target group(set of instances) Now create a service in the cluster with an ALB attached.

AWS EKS

AWS EKS EKS Overview Service managed by Amazon that allows us to run Kubernetes on AWS, without having to manage the infra Create a cluster eksctl create cluster \ --name my-cluster \ --region region-name \ --fargate ... [✓] EKS cluster "my-cluster" in "region-name" region is ready Above line tells the cluster is ready to use in the given region After successful creation of cluster, in ~/.kube/config we can see cluster details, like below - name: chiras@cisco.

AWS Certificate Manager

AWS certificate manager (ACM) Helps in managing SSL certificates Ways to renew certificates Buy and upload using AWS CLI ACM will provision and renew for free Integrations for which SSL certificates are loaded Load balancers Cloudfront distribution APIs on API gateways Creating SSL certificate Management console -> ACM Select public or private certificate Can import existing certificate or create new one Provide the domain name Choose validation criteria If CNAME is not created for the domain, create one Once the validation is completed, domain should be accessible using https request and certificate can also be viewed Pricing $400.

AWS Cloudfront

AWS CloudFront Overview Content deliver network that helps in improving read performance Content is cached 216 points of presence or edge locations, globally DDoS protection, integration with shield, web app firewall Can exppose https and talk to internal https endpoints Cloudfront- Origins S3 bucket Distributing files and caching at edge Enhanced security with origin access identity(OAI) used as ingress Custom origin ALB EC2 S3 website Cloudfront vs S3 cross origin Cloudfront is global, s3 co is to be done in every region Cloudfront caches for a TTL, S3 CO updates real-time CloudFront is suitable for static content, S3 CO for dynamic content to be available with low-latency in few regions CloudFront- S3 Creat bucket and add few objects Go to cloud front and create distribution with origin domain name as the bucket created Restrict bucket access - makes user have access only through CF URL and not Bucket URL Create or use an OAI Create distribution - takes sometime( around 10 mins) Now an OAI would have been created and if we go to the bucket-> bucket policy-> we can see only OAI created will have access Once CF is ready we can try to access the object, it’ll take time for CF to propogate to bucket(3-4 hours), till then we can make ACL and bucket ACL public and object as public, and now we can see the object CloudFront - Caching Caches based on- headers, session cookies, query string params Caching is based on TTL( 0 second to an year) CreateInvalidation API invalidates part of cache Because of caching if you change anything in an object and if the TTL is not crossed, the user will see older version of the object through CF Hence we can use CreateInvalidation for this problem CloudFront-> distribution-> invalidate-> create invalidation-> give the pattern of objects that’s to be invalidated (* for all objects) Now even if there is any change in any object of the bucket, changes will be reflected CloudFront - Security OAI for S3 Buckets HTTP, HTTPS, HTTP to HTTPS restrictions Geo restriction Can restrict based on location Whitelist and blacklist, both can be done use case- copyright laws for accessing content CF distribution-> restrictions-> enable geo restriction-> whitelist/blacklist regions CloudFront signed URL/ signed cookies If you want to distribute paid shared content to users signed URL/ cookie can attach policy like URL expiration IP ranges trusted signers Signed URL for individual files, cookies for multiple files(one cookie for many files) Pricing https://aws.

AWS Route53

AWS Route53 Overview Service in AWS that manages Domain name system(DNS) A domain costs around 12 dollars per year Amazon Route 53 effectively connects user requests to infrastructure running in AWS – such as Amazon EC2 instances, Elastic Load Balancing load balancers, or Amazon S3 buckets – and can also be used to route users to infrastructure outside of AWS Setting up a domain name :::info Route 53 is a global service.

AWS S3

AWS S3 Overview Allows users to store object in buckets(directories) Name should be unique globally buckets are define regionally Naming conventions No uppercase, underscore, 3-63 long, not an IP, must start with lowercase or number S3 objects Stored in s3://bucket-name/prefix/objectname prefix can be series of folders Max size of object 5TB Each object can have tags, version ID(if enabled) Creating a bucket and adding objects Management console -> Amazon S3 Create bucket Click on add objects and objects (folders can be created as well) Note that object can’t be viewed using the link to the path provided as the public access from bucket level and object level would have been blocked, changing the permissions will help us to see.

AWS VPC

AWS VPC fundamentals Overview VPC, Virutal private cloud is a private network to deploy our resources Through subnets we can partition our network inside a VPC public subnet is one which is accessible from the internet and private subnet is not accessible Route tables are used to define access from internet and between subnets Architecture of a VPC VPC is at region level - having multiple AZs An AZ has public and private subnets More on VPC Internet gateway helps VPC instances to connect to the internet Public subnet through internet gateway connects to internet Public subnet has route to IGW, private subnet has route to NAT gateway and instance Network ACL Acts as a firewall controlling traffic from and to subnet It can have allow or deny rules Attached at subnet level Rules can have only IP addresses Security groups Firewall controlling traffic to and from ENI or EC2 instance It can only have allow rules Rules can have IP addressed or other security groups VPC flow logs Captures information from VPC flow logs, subnet flow logs, ENI flow logs Helps to monitor and trouble shoot network issues Captures network information from managed services like RDS, Aurora and so on too Can store the logs to an S3 of be sent to cloudwatch logs VPC peering Connection two VPCs through AWS network For this IP address range on two VPCs should not be overlapping VPC connection is not transitive, it’ll only between two VPCs VPC endpoints Allows VPC to connect to AWS services using private network instead of public network They have advantage of low latency and secure connection VPC endpoint gateway for DynamoDB and S3 VPC endpoint interface for other services Site to site VPN and Direct connection We can connect our on-prem VPN to AWS using public encrypted network - Site to site VPN.