Hello Cloud Functions(python)

10.10.2021 — GCP, python — 3 min read

Cloud Functions run in a fully-managed, serverless environment where Google handles infrastructure, operating systems, and runtime environments completely on your behalf. Each Cloud Function runs in its own isolated secure execution context, scales automatically, and has a lifecycle independent from other functions.

Things to know before starting

There are two types of Cloud Functions,
- Google Cloud Functions, which allow you to run snippets of code in Google's infrastructure in response to events.
- Cloud Functions for Firebase, which triggers Google Cloud Functions based on events in Firebase (such as database or file writes, user creation, etc)
Functions can be triggered by HTTP, publishing a message to Pub/Sub topic, based on events in Firebase/Firestore, Cloud Scheduler, Cloud Storage. Functions can be categorized as two types, i) HTTP triggers and ii) Background functions.
Functions must be stateless -- one function invocation should not rely on in-memory state set by a previous invocation.
Each instance of a function handles only one concurrent request at a time.
If there is extremely rapid increase in inbound traffic can intermittently cause some requests to fail with an HTTP code of 500. This is because the requests are timing out in the pending queue while waiting for new instances to be created.
Cold starts may create latency problems this happens during new deployment and creating new instances during scaling.
Global Scope in a function is executed only once, when its first invoked. In a situation where the same funtion is reused for another execution, the variables you had set in global scope would not be set/triggered. So, write programs in such a way, it should not depend on global variables (or) reinitialize the variable at the end of execution.
At certain times, functions might be triggered more than once, your application should be able to handle that. You functions should be idempotent, an identical request can be made once or several times in a row with the same effect while leaving the server in the same state.
By default, a function times out after 1 minute, but you can extend this period up to 9 minutes(Add flag --timeout=540 during gcloud functions deploy)
/tmp/ path can be used for local storage during execution but be aware it takes up the space which you had allocated in memory default is 256MB. When CF completes its execution the files in /tmp/ are not deleted automatically. So, in a scenario where, if the same instance of the CF is used by the next request, memory can be used up by the files you had created in the previous innovation. Have a practice of cleaning up /tmp/ before program completion. To increase memory add flag --memory=512MB during gcloud functions deploy.
To share data between functions, use Firebase, Cloud Storage and Datastore do not use /tmp/.
When there is a version migration of any languages, you will be notified about it in a mail and there will be a mention of whether its going to be minor or major and a deadline will be given past that deadline, your function may be redeployed. If alls well, it will work, otherwise it will fail. So recommendation is, when you get a mail prepare for it. See the features of newer versions and google with keywords like "Migrating Cloud Functions Runtimes" and know the differences as some things might not be backward compatible.
- For NodeJS, GCP version migration happens in even number like NodeJS 12 -> NodeJS 14 -> NodeJS 16
Timestamps and Timezone used in Cloud Function is UTC. So if your program uses datetime, you need to make your program timezone aware.

Writing a Python Cloud Function

Python and Go Admin SDKs, all write methods are blocking. That is, the write methods do not return until the writes are committed to the database.

This reduces a lot of my code. Seriously, i was doing a lot to make things synchronous in my NodeJS Firebase CF's earlier.

Structuring your folder path

Below is the recommended folder path structure to create cloud function

1fn-function_name
2├── main.py             <-- Your cloud function's entrypoint should be defined in this file
3├── requirements.txt    <-- Specifies all the program dependencies
4└── localPkg/
5    ├── __init__.py     <-- adding this file turns localPkg folder into a module
6    └── myFunctions.py  <-- Contains your helper functions which is called from main.py in root directory.

Python Environment Setup

1# Environment Setup
2python3 -m venv env
3
4# Run everyday
5.\env\Scripts\activate
6set GOOGLE_APPLICATION_CREDENTIALS=<<path of the service account json file>>
7
8# Generate requirement.txt using below command
9pip3 freeze --all > requirements.txt

Simple Python Program Template

main.py

1# Imports from localPkg
2from localPkg.myFunctions import *
3
4import os
5import json
6import csv
7import sys, traceback
8from datetime import datetime
9from flask import escape
10
11# Imports Google Cloud Libraries
12# ....
13
14# [START hello-cf]
15def hello-cf(request):
16    """
17    This is a template for python cloud function with logging and global variables
18    Args:
19        request (flask.Request): The request object.
20    Returns:
21        The response text
22    """
23
24    # ---- Initializations ----
25    global global_arr
26
27    try:
28        # Extract value from queryString if any is passed
29        request_json = request.get_json(silent=True)
30        request_args = request.args
31
32        #date is queryString in http request
33        if request_json and 'date' in request_json:
34            paramDate = request_json['date']
35        elif request_args and 'date' in request_args:
36            paramDate = request_args['date']
37        else:
38            #Logic if no queryString is provided
39            # ....
40            print("")
41            
42    except ValueError as ve:        
43        # Structured Log Entry using print
44        entry = dict(
45            severity="INFO",
46            message='All done',
47            # Log viewer accesses 'component' as jsonPayload.component'
48            log_entries={ index : item for index,item in enumerate(global_arr) },
49        )
50        print(json.dumps(entry))
51        global_arr = []
52
53        return str(ve)
54
55    except Exception as e:
56        print("Unexpected ERROR: ", sys.exc_info())        
57        er_str = ''
58        for el in traceback.format_exc().splitlines():
59            print(el)
60            er_str=er_str+'::'+el
61        global_arr.append(er_str)
62        global_arr.append("return=no data")
63
64        # Structured Log Entry using print
65        entry = dict(severity="ERROR", message=er_str 
66                    , log_entries={ index : item for index,item in enumerate(global_arr) }, )
67        print(json.dumps(entry))
68        
69        global_arr = []
70        return 'no data'
71    else:
72        # Structured Log Entry using print
73        entry = dict( severity="INFO", message='All done'
74                    , log_entries={ index : item for index,item in enumerate(global_arr) }, )
75        print(json.dumps(entry))
76        global_arr = []
77
78        return 'success'
79# [END hello-cf]

Below is the code in sub-folder localPkg

./localPkg/__init__.py

1from .myFunctions import *

This is the myFunctions module

./localPkg/myFunctions.py

1# Imports
2#...
3
4# Initialize Global Variables
5global_arr = []
6
7#...

Sample Python Cloud Function Program

This cloud function does the following,

Download the bhavcopy file from Bombay Stock Exchange to cloud functions local folder /tmp/
Unzips the file and extracts the .CSV file
Uploads both the files to Google Cloud Storage

Folder structure

1fn-function_name
2├── main.py
3├── requirements.txt
4└── localPkg/
5    ├── __init__.py
6    └── myFunctions.py

main.py

1# Imports from localPkg
2from localPkg.myFunctions import *
3
4import os
5import json
6import csv
7import sys, traceback
8from datetime import datetime
9from flask import escape
10
11# [START download_bhavcopy]
12# def download_bhavcopy():
13def download_bhavcopy(request):
14    """
15    This is a template for python cloud function with logging and global variables
16    Args:
17        request (flask.Request): The request object.
18    Returns:
19        The response text
20    """
21
22    # ---- Initializations ----
23    global global_arr
24    
25    try:
26        # ---- Getting Dates from QueryString ----
27        request_json = request.get_json(silent=True)
28        request_args = request.args
29
30        if request_json and 'date' in request_json:
31            fromDate = request_json['date']
32        elif request_args and 'date' in request_args:
33            fromDate = request_args['date']
34        else:
35            # If there is no queryString. Current date will be considered
36            fromDate = datetime.date(datetime.now(timezone('UTC')).astimezone(timezone('Asia/Calcutta'))).strftime("%Y%m%d")
37
38        global_arr.append('fromDate='+fromDate)
39
40        # fromDate = '20210901'
41        bucket = '<<bucket-name>>'
42        csvPath = 'bse/bhavcopy_csv/'
43        
44        # ---- Cloud Functions temporary folder ---- 
45        temp_folder = '/tmp/'
46
47        bDate = convert_date(fromDate, '%d%m%y')
48        url_bhavcopy = 'https://www.bseindia.com/download/BhavCopy/Equity/EQ_ISINCODE_<<date>>.ZIP'.replace('<<date>>',bDate)
49        csvFile = 'EQ_ISINCODE_<<date>>.CSV'.replace('<<date>>',bDate)
50
51        # ---- Download the bhavcopy ----
52        bhav_file =  download_to_tmp(temp_folder, url_bhavcopy)
53        if(bhav_file == ''):
54            raise ValueError(bhav_file)
55
56        # ---- Unzipping file in Cloud Functions /tmp/ folder ----
57        bhav_file = temp_folder+bhav_file
58        bhav_file = temp_folder+unzip_file(temp_folder, bhav_file, csvFile)
59
60        # ---- PreProcess CSV save as new file ----
61        ppCSV = temp_folder+'pp-'+csvFile
62        with open(bhav_file, 'r') as inf, open(ppCSV, 'w', newline='') as of:
63            r = csv.reader(inf, delimiter=',')
64            w = csv.writer(of, delimiter=',')
65            total_rows = 0
66            for line in r:
67                total_rows+=1
68                trim = (field.strip() for field in line)
69                w.writerow(trim)            
70
71        # ---- Upload to GCS CSV & Preprocessed file ----
72        temp = csvPath+bhav_file.split('/')[-1]
73        upload_blob(bucket, bhav_file, temp)
74
75        temp = csvPath+ppCSV.split('/')[-1]
76        upload_blob(bucket, ppCSV, temp)
77
78    except Exception as e:
79        print("Unexpected ERROR: ", sys.exc_info())        
80        er_str = ''
81        for el in traceback.format_exc().splitlines():
82            print(el)
83            er_str=er_str+'::'+el
84        global_arr.append(er_str)
85        global_arr.append("return=no data")
86
87        # Structured Log Entry using print
88        entry = dict(severity="ERROR", message=er_str
89                    , log_entries={ index : item for index,item in enumerate(global_arr) }, )
90        print(json.dumps(entry))
91        
92        global_arr = []
93        return 'no data'
94
95    else:
96
97        # Structured Log Entry using print
98        entry = dict( severity="INFO", message='All done'
99                    , log_entries={ index : item for index,item in enumerate(global_arr) }, )
100        print(json.dumps(entry))
101        global_arr = []
102        return 'success'
103# [END download_bhavcopy]
104
105
106# if __name__ == "__main__":
107#     download_bhavcopy()

__init__.py

1# Added . as python not able to find the myFunctions file
2from .myFunctions import *

./localPkg/myFunctions.py

1import os
2import sys
3import urllib.request 
4import socket
5import zipfile
6import time
7from datetime import datetime
8from pytz import timezone
9
10# Importing Google libraries
11from google.cloud import storage 
12
13# Global Variables
14global_arr = []
15
16### Functions
17# Download the file in the URL to the specified folder path 
18def download_to_tmp(temp_folder, url):
19    global global_arr
20    # print('Entry', global_arr)
21
22    try:        
23        socket.setdefaulttimeout(20)
24        opener = urllib.request.build_opener()
25        opener.addheaders = [('User-agent', 'Mozilla/5.0')]
26        urllib.request.install_opener(opener)
27
28        os.makedirs(os.path.dirname(temp_folder), exist_ok=True)
29        osfp_folder = os.path.join(temp_folder)
30        temp = 'Downloading '+ url + ' --> ' + osfp_folder+url.split('/')[-1]
31        global_arr.append(temp)
32        print(temp)
33
34        filename = url.split('/')[-1]
35        print(osfp_folder + filename)
36        resultFilePath, responseHeaders = urllib.request.urlretrieve(url, osfp_folder + filename)
37
38    except Exception as e:
39        filename = ''
40
41    return filename
42
43# Unzips the file
44def unzip_file(temp_folder, fp, fn):
45    with zipfile.ZipFile(fp) as z:
46        with open(temp_folder + fn, 'wb') as f:
47            f.write(z.read(fn))
48    return fn
49
50# Standard function to upload file to GCS
51def upload_blob(bucket_name, source_file_name, destination_blob_name):    
52    global global_arr
53
54    # If you don't specify credentials when constructing the client, the
55    # client library will look for credentials in the environment.
56    storage_client = storage.Client() 
57    bucket = storage_client.bucket(bucket_name)
58    blob = bucket.blob(destination_blob_name)
59 
60    blob.upload_from_filename(source_file_name)
61 
62    temp = "Uploaded {} to {} successfully.".format(source_file_name, destination_blob_name)
63    # print(temp)
64    global_arr.append(temp)
65    return None
66
67# Converts date to the format specified in the parameter
68def convert_date(parmDate, dtFormat):
69    return datetime.strptime(parmDate, '%Y%m%d').strftime(dtFormat)

Deploying the Cloud Function

1gcloud config set project <<project-name>>
2
3# Using ^ as i am submitting from Windows
4gcloud functions deploy download_bhavcopy ^
5     --runtime python39 ^
6     --trigger-http ^
7     --allow-unauthenticated ^
8     --timeout=540s
9
10# Following is the output
11Deploying function (may take a while - up to 2 minutes).../
12For Cloud Build Logs, visit: https://console.cloud.google.com/cloud-build/builds;region=us-central1/9cd4f07f-3b73-4f2e-9c2e?project=254912435
13Deploying function (may take a while - up to 2 minutes)...done.
14availableMemoryMb: 256
15buildId: 9cd4f07f-3b73-4f2e-9c2e
16buildName: projects/254912435/locations/us-central1/builds/<<some-hash>>
17entryPoint: download_bhavcopy
18httpsTrigger:
19  securityLevel: SECURE_OPTIONAL
20  url: https://us-central1-<<project-name>>.cloudfunctions.net/download_bhavcopy
21ingressSettings: ALLOW_ALL
22labels:
23  deployment-tool: cli-gcloud
24name: projects/<<project-name>>/locations/us-central1/functions/download_bhavcopy
25runtime: python39
26serviceAccountEmail: <<project-name>>@appspot.gserviceaccount.com
27sourceUploadUrl: https://storage.googleapis.com/gcf-upload-us-central1-<<some-hash>>/57467b94-d75a-4f74-bb26-b22be872ae72.zip
28status: ACTIVE
29timeout: 540s
30updateTime: '2021-10-12T15:06:55.539Z'
31versionId: '1'

Above CF can be trigged like from the browser

1https://us-central1-<<project-name>>.cloudfunctions.net/download_bhavcopy?date=20210902
2(or
3https://us-central1-<<project-name>>.cloudfunctions.net/download_bhavcopy