Showing posts with label aws. Show all posts
Showing posts with label aws. Show all posts

2023-04-18

How to use DNS SDK in Golang

So we're gonna try to manipulate DNS records using go SDK (not REST API directly). I went through first 2 page of google search results, and companies that providing SDK for Go were:

  1. IBM networking-go-sdk - 161.26.0.10 and 161.26.0.11 - timedout resolving their own website
  2. AWS route53 - 169.254.169.253 - timedout resolving their own website
  3. DNSimple dnsimple-go - 162.159.27.4 and 199.247.155.53 - 160-180ms and 70-75ms from SG
  4. Google googleapis - 8.8.8.8 and 8.8.4.4 - 0ms for both from SG
  5. GCore gcore-dns-sdk-go - 199.247.155.53 and 2.56.220.2 - 0ms and 0-171ms (171ms on first hit only, the rest is 0ms) from SG

I've used google SDK before for non-DNS stuff, a bit too raw and so many required steps. You have to create a project, enable API, create service account, set permission for that account, download credentials.json, then hit using their SDK -- not really straightforward, so today we're gonna try G-Core's DNS, apparently it's very easy, just need to visit their website and sign up, profile > API Tokens > Create Token, copy it to some file (for example: .token file).

This is example how you can create a zone, add an A record, and delete everything:

 package main

import (
  "context"
  _ "embed"
  "strings"
  "time"

  "github.com/G-Core/gcore-dns-sdk-go"
  "github.com/kokizzu/gotro/L"
)

//go:embed .token
var apiToken string

func main() {
  apiToken = strings.TrimSpace(apiToken)

  // init SDK
  sdk := dnssdk.NewClient(dnssdk.PermanentAPIKeyAuth(apiToken), func(client *dnssdk.Client) {
    client.Debug = true
  })
  ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
  defer cancel()

  const zoneName = `benalu2.dev`

  // create zone
  _, err := sdk.CreateZone(ctx, zoneName)
  if err != nil && !strings.Contains(err.Error(), `already exists`) {
    L.PanicIf(err, `sdk.CreateZone`)
  }

  // get zone
  zoneResp, err := sdk.Zone(ctx, zoneName)
  L.PanicIf(err, `sdk.Zone`)
  L.Describe(zoneResp)

  // add A record
  err = sdk.AddZoneRRSet(ctx,
    zoneName,        // zone
    `www.`+zoneName, // name
    `A`,             // rrtype
    []dnssdk.ResourceRecord{
      {
// https://apidocs.gcore.com/dns#tag/rrsets/operation/CreateRRSet
        Content: []any{
          `194.233.65.174`,
        },
      },
    },
    120, // TTL
  )
  L.PanicIf(err, `AddZoneRRSet`)

  // get A record
  rr, err := sdk.RRSet(ctx, zoneName, `www.`+zoneName, `A`)
  L.PanicIf(err, `sdk.RRSet`)
  L.Describe(rr)

  // delete A record
  err = sdk.DeleteRRSet(ctx, zoneName, `www.`+zoneName, `A`)
  L.PanicIf(err, `sdk.DeleteRRSet`)

  // delete zone
  err = sdk.DeleteZone(ctx, zoneName)
  L.PanicIf(err, `sdk.DeleteZone`)
}

The full source code repo is here. Apparently it's very easy to manipulate DNS record using their SDK, after adding record programmatically, all I need to do is just delegate (set authoritative nameserver) to their NS: ns1.gcorelabs.net and ns2.gcdn.services.

In my case because I bought the domain name on google domains, then I just need to change this: 

 
Then just wait it to be delegated properly (until all DNS servers that still caching the old authorized NS cleared up), I guess that it.

2021-12-18

Coolest PaaS/IaaS I've ever use: Jelastic

So, I'm looking an simplest deployment strategy for my next side project, I don't want to use Kubernetes since I'm all alone XD, learning Nomad, WayPoint, Swarm, and other popular tool to make it easy like Portainer, but why they doesn't make it just as simple as Vercel or Fly.io). Also don't want to use big cloud providers (GCP, AWS, Azure, etc) which the UI quite sucks like everything developed by different team with lack of communication and you have to do a lot of setup hassle just to deploy simple things. Then I found a really cool product called Jelastic, that fit my needs:
  1. Can autoscale out (like AWS ELB/ECS, GCR, ACS, etc) and auto-clustering (as easy as CloudSQL or AWS RDS/Aurora, but can be automatic)
  2. Can autoscale up '__') without downtime, only took 1 second to scale up from 1 core 640MB to 16 core 32GB (seems like they only changing container's resource quota limit) but you can see the changes directly without restart
  3. Can deploy VPS on the same cluster/network (for my databases, since I don't use "standard/popular" databases) and it's super cheap (it only took 3.9$ per month to deploy a VPS with 1 static IP, and can autoscale up), you only need to pay what you utilize (CPU and RAM usage), not charged 100% when server up unlike other VPS providers
  4. The UI doesn't sucks XD you can WebSSH, normal SSH (as long as have real IP), easy SSL setup, super easy to change config, the lacking part about Jelastic probably configfile/gitops-based setup (for working with multiple members in the future) at least there's API and CLI to create and modify environment, not sure if there's auditing available (haven't checked yet). 
  5. Can also deploy automatically from git (checked every N minutes) or CI pipeline or using CLI.
  6. Easy to move (live migration) to different providers, change ownership of a cluster, or if it's not enabled, at least there's no vendor locking, you can also manually export and import environment (for example copying staging setup to production has similar architecture just different deployment branch and scaling strategy).


Other cool things that I won't use: deploying any-container/docker-based with easy steps, deploying kubernetes, bunch of stack in marketplace provided (may vary on different provider).

For 3.9$ (if you utilize only 1%) per month (16 core, 32GB RAM, 200 GB NVMe VPS, 1 static IP, provider: ToggleBox), you can get the greenest (highest on average) result among all VPS I've ever tried:


You can see the raw benchmark result here and recap here.

What's the catch?
  1. It's quite expensive if you utilize 100% (around 339$ if you use ToggleBox for the specs above), for comparison:
    1. cheapest highest spec Contabo's VPS (9 core, 60GB RAM, 1.5TB SSD) unmetered bandwidth only cost $55-ish per month (not apple-to-apple since it's different spec and performance, also this is what you should pay per month regardless your utilization)
    2. similar spec GCE n1-custom-16-32768 (16 core, 32GB, 200GB SSD) non-committed, cost $525 excluding bandwidth
    3. similar spec AWS EC2 a1.xlarge (16 core, 32GB RAM, 200GB gp2 SSD) on-demand, only cost  $317 excluding bandwidth
    4. similar spec Azure F16s (16 core, 32GB RAM, 256GB SSD) pay-as-you-go, cost $634 excluding bandwidth
    5. cheapest OVH on SG (8 core. 64GB RAM, 400GB SSD) only cost $135 with unmetered 200Mbps bandwidth
    but still, this is way cheaper for minimum usage than if you use GCR you will be billed around ~$10 per month for idle instance, or ~$37 for standby instance (for 1 VCPU, 1 GB RAM, not including bandwidth that quite pricey $0.085)
  2. Some provider have different "free" tier, for example ToggleBox give free 2GB bandwidth per hour (GCR only give free 1GB per month XD), some other provider give free 1 static IP, some other provider give free 10GB disk usage per hour, etc.
  3. License might be pricey if you install it on your own cluster instead of using the already provided (eg. DewaCloud or CloudKilat for Indonesia region, ToggleBox for US region, etc), but they have profit sharing model if you are a reseller (have your own VPS and rent it).
  4. The billing is hourly (so you will always billed at minimum 1 cloudlet -- specs of 1 cloudlet can be vary per provider), compared to for example GCR that use second as minimum billing resolution (VCPU, GB RAM, Requests, and Bandwidth).


That's it for now, I'll create a new post if I found something better.

2021-08-05

You don't need Kubernetes

Everyone using kubernetes, even they don't have to, which adds additional complexity. In this article we will discuss when we need Kubernetes, and when we don't, and what's the alternative?

When you need Kubernetes?
  1. If you have lot of teams (>3, not 3 person, but at least 3 teams) and services, it's better to use Kubernetes or any other orchestration service to utilize your servers
    Why? because that way they can deploy each services independently, and service mesh management is far easier if using orchestration.
    Alternative? use Nomad or standard rsync with autoreloader (eg. overseer if you are using Golang) or if you are using dynamic languages (or old CGI/FastCGI) that are not creating own web server usually you don't need an autoreloader since source code on the server anyway and will be autoreloaded by the web server
  2. If you have more than 3+ powerful servers, it's better to use Kubernetes or any other orchestration service 
    Why? with orchestration service we can binpack and fully utilize those servers better than creating bunch of VMs inside the servers
    Alternative? use Ansible to setup those servers, or Nomad to orchestate, or create a script to rsync to multiple servers
  3. If you need canary or A/B deployment
    Why? creating canary-like deployment manually (setting up load balancer to send a percentage of request to specific nodes is a bit chore)
    Alternative? use Nomad and Consul service registry
  4. When you are using slow programming language implementation that the first bottleneck is not database but your API/Web requests implementation
    Why? normally if you code using compiled programming language, the bottleneck is not the network/number of hits, but the database, which usually solved with caching if it just read, or CDN if static files. But when you are using slow programming languages (eg. Ruby, Python, PHP5, etc), you mostly will hit bottleneck on the API/Web requests side first before the database. So you'll need to spawn additional pod to handle those requests.
    Alternative? Amazon AutoScaling, Google Cloud AutoScaling Group, Azure AutoScale, or just do a capacity planning and scale up/out manually before the peak season (eg. Black Friday, Christmas,  discount day, etc) and scale down/in after peak season ends.
When you don't need Kubernetes?
  1. When you only have a little <=3 or bunch of least powerful servers that are utilized well.
    Seriously you don't need it, just a simple rsync script (either manual or in CI/CD pipeline) or Ansible script would be sufficient
  2. When you are alone or in a small team (no other team), writing modular monolith is fine until it's painful to build, test, and deploy, then that's the moment that you will need to split to multiple services.
  3. When you only have production and test environment without A/B or canary deployment.
    Again, Kubernetes will be totally overkill in this case.
    1. always cache reads
    2. prepare for nth server when it's already 70% load (you need to provision a new node anyway if kubernetes also hit the load), either manually or use cloud providers' autoscaler
    3. put slow tasks/computation as background job, so web services could always respond fast. Even when you have autoscaler for APIs/web services, if the bottleneck is the database or I/O (which is more likely than the processor/computation), it would be totally useless.
  4. When you don't want Kubernetes to be the 30% overhead your servers, use lightweight orchestration and service discovery instead, like Nomad with Consul.
Well, for the last part, you are mostly don't need Kubernetes, don't be eaten by the hype, especially if you deploy databases on the pod, it's unstable AF, you don't need to automatically scale out/in the databases anyway (usually scale up if the database product is quite hassle to scale out), it would be a crazy thing to move around gigabytes of data because the automatic scaling. I hope this article could help people that started to build their startup to start provision with proper capacity planning and releasing instead of prematurely optimize their architecture and infrastructure with Kubernetes.

FAQ

So, what are your qualification saying this?
I worked for small companies (<7 engineers) and huge companies (700+ engineers) that are both using Kubernetes, and I realized the difference why one is a must, and why the other one is just decision based on hype.

EDIT

I found something that even simpler than nomad/waypoint/kubernetes, jelastic! see the demo video here.

2016-02-13

Cloud Computing Platform

So recently my colleague's startup really need a scalable application, that could handle billion hits, so I really need to learn to utilize cloud service instead of depend on one or two normal PC with huge RAM (that I prefer to use when the users are lower than 10k of non-static hit), rather than building your own server farm, that probably costs higher. Some of the famous cloud (IaaS or PaaS) services are:
Some parts of the information above shamelessly copied from wikipedia. There are some comparison about the price, more price, even more price, featuresterms, and performance out there. Which one the best for you? depends on your budget and your developer's skill.