Access file absolute path in Serverless Custom App

Hello Everyone! Hope you all are good.

I’m developing a serverless custom app in freshsales suite and in there, I need to send logs to Google Cloud Logging, for this I’m using the Cloud Logging Library for Nodejs.

The issue is, to use this library outside of Google Cloud Plataform a need to use an API key (provided by GCP), that consists of a json file, in order authenticate the requests.

To inicialize the client I need to provide the API key path:

image

That variable filePath is defined as follows:

image

When I test locally, it works as expected, the logs are created, but when I deploy the app in the crm, the logs are not created in GCP, and the filePath returns /var/task/developer/server/lib/config/APIKEY.json. This is probably not the right path to the file…

So, how can I access this file correctly in the serverless custom app?

Thanks!

@Matheus_Souza_Silva

What helped you conclude —

The path that is resolved is the wrong path?

Can you try using require() to load the JSON file with a relative file path? For example, if the JSON file is in ./server/lib/config/APIKEY.json and you are accessing it from ./server/server.js, then use:

const credentials = require("./lib/config/APIKEY.json");
1 Like

@Saif I think it is the wrong path because the log wasn’t created in GCP, if I run the app locally, the log is created with the file path provided by my computer. The only difference of the app running locally and in production is the file path, so that is why I think the path that process.cwd() provides when running in production is invalid.

Do you know the absolute path of the server folder when the custom app is deployed? It would help immensely.

@kaustavdm When I try using require(), the following error occurs:

Error: Cannot find module "./config/APIKEY.json"

I think this happens because APIKEY.json is not a module, it’s just a json file. I need to provide the file path itself.

One other thing I forgot to mention, is that I use this library in a separate file, I export the functions that I want and import in server.js using require(./lib/logger.js), in there I need to access my credentials file path. The logger.js file is located in the lib folder, but I think this doesn’t change much what I’ve already said :laughing:.

Just for clarification, process.cwd() provides var/task/developer, I want to be able to access the server folder.

1 Like

I have the same intuition as @kaustavdm in this case. I hoped require(..) might have worked.

Requesting help from @App-Platform-Squad

Looks like @Matheus_Souza_Silva is trying to get data in a credentials.json file into server.js at runtime. However, process.cwd() in production resolves differently to what’s in local, and using require(..) doesn’t treat credentials.json as a JSON. Any alternative suggestions?

1 Like

This is curious. I suspect it has something to do with how Lambda works. I also wonder if our platform runtime overloads the require() function. Let me try to see if I can make it work.

1 Like

Hey @Matheus_Souza_Silva

I would not rely on absolute paths within a serverless app. For security reasons, we will block access to the filesystem within a serverless execution. Do you mind trying with a relative path instead?

Meanwhile, I think we have other example apps that might be trying to do this. I will request @Saif to check them out and get back to you.

2 Likes

Hello everyone!

I’ve tested with a relative path but, it didn’t work…

Any updates @satwik?

There are couple of things I am able to identify

Sounds like this could actually be an important detail.
Do you mind sharing a zip file? You can remove/minify any irrelevant code. I am importantly trying gauge at directory structure.

cc: @Matheus_Souza_Silva

Hi there! Thanks for replying.

I’m sending you the zip file, there I just simply put the folder structure that is relevant, The important thing is just to get the correct file path.

testFilePath.zip (1.7 KB)

Thank you!

1 Like

I tried publishing the app and see the logs:

I could see the path that you mentioned. I also couldn’t access the credentials.json file using require(..) as you described.

See Error Stack Trace
Inicializing logging...
/Users/saifas/MyLab/fapps/teste logger/server/server/lib/config/jsonKey.json
/usr/local/lib/node_modules/fdk/lib/event_handler/framework.js:105
            throw createModuleError(relativePath);
            ^

Error: Cannot find module “./lib/config/jsonKey.json”
at createModuleError (/usr/local/lib/node_modules/fdk/lib/event_handler/framework.js:27:17)
at require (/usr/local/lib/node_modules/fdk/lib/event_handler/framework.js:105:19)
at server.js:140:41
at server.js:152:7
at Script.runInContext (vm.js:144:12)
at ProductEvent.sandboxExecutor (/usr/local/lib/node_modules/fdk/lib/event_handler/framework.js:225:25)
at handler (/usr/local/lib/node_modules/fdk/lib/event_handler/framework.js:361:31)
at handleRequest (/usr/local/lib/node_modules/fdk/lib/routes/beevents.js:108:3)
at Layer.handle [as handle_request] (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/layer.js:95:5)
at next (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/route.js:144:13)
at Route.dispatch (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/route.js:114:3)
at Layer.handle [as handle_request] (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/layer.js:95:5)
at /usr/local/lib/node_modules/fdk/node_modules/express/lib/router/index.js:284:15
at Function.process_params (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/index.js:346:12)
at next (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/index.js:280:10)
at Function.handle (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/index.js:175:3)
at router (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/index.js:47:12)
at Layer.handle [as handle_request] (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/index.js:328:13)
at /usr/local/lib/node_modules/fdk/node_modules/express/lib/router/index.js:286:9
at Function.process_params (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/index.js:346:12)
at next (/usr/local/lib/node_modules/fdk/node_modules/express/lib/router/index.js:280:10)

It ideally should be possible in Node runtime. There’s a possibility of deviation from original behavior.

File Structure
❯ tree
.
├── config
│   └── iparams.json
├── coverage
│   ├── base.css
│   ├── block-navigation.js
│   ├── favicon.png
│   ├── index.html
│   ├── prettify.css
│   ├── prettify.js
│   ├── server
│   │   ├── index.html
│   │   ├── lib
│   │   │   ├── index.html
│   │   │   └── logger.js.html
│   │   └── server.js.html
│   ├── sort-arrow-sprite.png
│   └── sorter.js
├── dist
│   └── teste\ logger.zip
├── log
│   └── fdk.log
├── manifest.json
└── server
    ├── lib
    │   ├── config
    │   │   └── jsonKey.json
    │   └── logger.js
    ├── node_modules
    │   ├── inherits
    │   │   ├── LICENSE
    │   │   ├── README.md
    │   │   ├── inherits.js
    │   │   ├── inherits_browser.js
    │   │   └── package.json
    │   ├── path
    │   │   ├── LICENSE
    │   │   ├── README.md
    │   │   ├── package.json
    │   │   └── path.js
    │   ├── process
    │   │   ├── LICENSE
    │   │   ├── README.md
    │   │   ├── browser.js
    │   │   ├── index.js
    │   │   ├── package.json
    │   │   └── test.js
    │   └── util
    │       ├── LICENSE
    │       ├── README.md
    │       ├── package.json
    │       ├── support
    │       │   ├── isBuffer.js
    │       │   └── isBufferBrowser.js
    │       └── util.js
    ├── package-lock.json
    ├── server.js
    └── test_data
        └── onContactCreate.json

I’ve dropped you a private message to collect app specific details.

@Matheus_Souza_Silva

We are monitoring your app in production

Do mind invoking the app again to reproduce failing to access credentials? Let us also know dates and period of time when you reproduce it…

Hi @Matheus_Souza_Silva ,

I’d faced the same issues when I wanted to use Google Cloud Logging.
I gave up trying to read the json file after several attempts and found a way around it.

I created a helper class that would wrap the Goggle Cloud Logging logic inside it without the need for an external json file.

Note that the below code is in TypeScript, so it might contain type definitions along as well.

Define your GoogleCloudLogging class

// GoogleCloudLogging.ts
import { Logging } from '@google-cloud/logging';
import { ApiResponse } from '@google-cloud/logging/build/src/log';

export interface GoogleServiceAccountConfig {
  client_email: string;
  private_key: string;
  project_id: string;
}

export type AnyJson = boolean | number | string | null | JsonArray | JsonMap;
export interface JsonMap {
  [key: string]: AnyJson;
}
export type JsonArray = Array<AnyJson>;

export interface GoogleCloudLoggingConfig extends GoogleServiceAccountConfig {
  logName: string;
}

export enum LogSeverity {
  ALERT = 'ALERT',
  CRITICAL = 'CRITICAL',
  DEBUG = 'DEBUG',
  DEFAULT = 'DEFAULT',
  EMERGENCY = 'EMERGENCY',
  ERROR = 'ERROR',
  INFO = 'INFO',
  NOTICE = 'NOTICE',
  WARNING = 'WARNING',
}

export class GoogleCloudLogging {
  private logging: Logging;

  constructor(private config: GoogleCloudLoggingConfig) {
    this.logging = new Logging({
      credentials: {
        client_email: this.config.client_email,
        private_key: this.config.private_key,
      },
      projectId: this.config.project_id,
    });
  }

  log(jsonMap: JsonMap, severity: LogSeverity): Promise<ApiResponse> {
    const log = this.logging.log(this.config.logName);

    const metadata = {
      resource: {
        type: 'global',
      },
      severity,
    };

    const entry = log.entry(metadata, jsonMap);

    return log.write(entry);
  }
}

Usage

The credentials you require from GCP are client_email, private_key, and project_id. You could store them in a separate secrets.ts file and export it only to be imported wherever you want, just like any Javascript module, thereby obviating the need for a json file.

Here’s how you’d use the class defined above.

// Step 1: Import the module
// server.ts
import {
  JsonMap,
  LogSeverity,
  GoogleCloudLogging,
} from './GoogleCloudLogging';

// Step 2: Create an instance of GoogleCloudLogging by giving it the necessary credentials
const googleCloudLogging = new GoogleCloudLogging({
  client_email: '<gcp-client-email>',
  private_key: '<gcp-private-key>',
  project_id: '<gcp-project-id>',
  logName: '<your-app-name>'
});

// Step 3: Use the log method to send any payload of your choice. Remember to set the severity as well.
googleCloudLogging.log({
  {
    ...payload // Your payload goes here
  },
  severity: LogSeverity.DEFAULT
});

Update manifest.json

Make sure to include @google-cloud/logging as a dependency in your manifest.json.

//manifest.json
{
  "platform-version": "2.2",
  "dependencies": {
    "@google-cloud/logging": "9.5.1"
  },
  ... // rest of the stuff
}

Closing notes

This approach has been successfully running for several years now on several of my apps in the Freshworks Marketplace.

Remember to not commit your secrets.ts file in your revision control system as it could lead to a security vulnerability.

Hope this helps.

Regards,
Arun Rajkumar

cc: @Saif , @satwik , @Raviraj

3 Likes

What you described is a very clever approach to the solution, Arun. Do you think we can use something like secrets.js alongside classes (credentials in the constructor or something) as a possible solution? Just curious to know if typescript makes any unique contribution to solving the problem or if it is just a preference.

@Matheus_Souza_Silva - you seem to have reproduced the issue in production. Would you mind sharing with us an approximate period?

1 Like

Hi @Saif, I’ve triggered the app, the period is beetween 18/08 09:30am to 09:40am Brazilian time. Thanks!

@arunrajkumar235 I’ll try to implement your solution, thanks for sharing!

1 Like

I prefer keeping the secrets.js as a separate file as I can add it to .gitinore. If you keep the secrets in the same file as the constructor, it would be hard to separate data from code. That way you don’t have to commit your secrets.js file ever.

What is TypeScript

TypeScript is merely a superscript of Javascript. NodeJS doesn’t understand TypeScript. So, we need to run the TypeScript compiler which will take a .ts file as input and emit a .js file. The compiler comes with several options to target the ES version. For example, you could write your code in TypeScript making use of the latest ES2022 features and the compiler could be configured to output code in ES5 compatibility.

Why use TypeScript

The “why” of using TypeScript is for type safety. For example, in Javascript you could define a variable and assign values of any type during run-time. This leads to possible inconsistencies and bugs in run-time.

let a = 5;
a = "Arun";
a = {
  name: "Arun",
  place: "Chennai",
};

Such a code would be a nightmare to debug. TypeScript helps by performing a static analysis of types while you are coding in your IDE itself with the help of extensions.

TypeScript helps catch bugs even before occur. It’s like having a senior developer peering through your code every time you write code.

When adopted in its true spirit, it can be very beneficial especially when your application grows large in size.

2 Likes

Hi everyone! Hope you’re all doing alright.

@arunrajkumar235 , I’m trying to implement what you suggested, and I’ve some questions.

In the serverless custom app there is the server.js, and you have this file in typescript, I tried using the file in the same extension, but it didn’t work. How you did it?

Also, you use Ecmascript’s modules and for me, the fdk only accepts CommonJS’s modules. How you use it?

Because of those questions, I opted to convert your approach, to javascript instead, and when I tested locally, it worked, but when a deployed the app in CRM, it doesn’t, so I really don’t know what to do next :smiling_face_with_tear: .

hi @Matheus_Souza_Silva ,

Wanted to clarify certain things.

  • NodeJS cannot interpret TypeScript as it only understands Javascript.
  • You will need to use a TypeScript compiler to get the JS files in ES5/6 or whatever is supported by Freshworks Serverless.
  • You can specify the module dependency option in the TypeScript compiler options file.
  • You can use an online tool to compile TypeScript to Javascript.

CommonJS module in Javascript

Here’s the TypeScript code that was compiled into Javascript ES2015 with CommonJS.

"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.GoogleCloudLogging = exports.LogSeverity = void 0;
// GoogleCloudLogging.ts
const logging_1 = require("@google-cloud/logging");
var LogSeverity;
(function (LogSeverity) {
    LogSeverity["ALERT"] = "ALERT";
    LogSeverity["CRITICAL"] = "CRITICAL";
    LogSeverity["DEBUG"] = "DEBUG";
    LogSeverity["DEFAULT"] = "DEFAULT";
    LogSeverity["EMERGENCY"] = "EMERGENCY";
    LogSeverity["ERROR"] = "ERROR";
    LogSeverity["INFO"] = "INFO";
    LogSeverity["NOTICE"] = "NOTICE";
    LogSeverity["WARNING"] = "WARNING";
})(LogSeverity = exports.LogSeverity || (exports.LogSeverity = {}));
class GoogleCloudLogging {
    constructor(config) {
        this.config = config;
        this.logging = new logging_1.Logging({
            credentials: {
                client_email: this.config.client_email,
                private_key: this.config.private_key,
            },
            projectId: this.config.project_id,
        });
    }
    log(jsonMap, severity) {
        const log = this.logging.log(this.config.logName);
        const metadata = {
            resource: {
                type: 'global',
            },
            severity,
        };
        const entry = log.entry(metadata, jsonMap);
        return log.write(entry);
    }
}
exports.GoogleCloudLogging = GoogleCloudLogging;