Skip to content

database is not create for instance when using centralized infra #12

Open
Open
database is not create for instance when using centralized infra#12
Assignees
zelima

Description

@zelima
zelima
opened on Nov 21, 2019

As a cco user, I've tried to deploy a CKAN instance test with centralized infra but I don't see the databse test created there. This leads to the authentication Error when CKAN is trying to start and connect to DB. (See analysis for more info)

Acceptance Criteria

  • CKAN Instance has it's dedicated database in centralized DB
  • CKAN can connect to DB

Tasks

  • Find where the passwords are created for CKAN (as I see DB URLs)
  • Find the part responsible for creating databases
    • Why that part is not executed?
  • Refactor if reasonable and make it happen

Analysis

This is all I see in database instance

psql postgresql://postgres:123456@db.ckan-cloud
\l
                                    List of databases
       Name       |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
------------------+----------+----------+------------+------------+-----------------------
 ckan             | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =Tc/postgres         +
                  |          |          |            |            | postgres=CTc/postgres+
                  |          |          |            |            | ckan=CTc/postgres
 postgres         | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0        | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                  |          |          |            |            | postgres=CTc/postgres
 template1        | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                  |          |          |            |            | postgres=CTc/postgres
 template_postgis | postgres | UTF8     | en_US.utf8 | en_US.utf8 |

Funny thing is that database credentials (User, passwords) are set but the are not actually created

cat /etc/ckan-conf/secrets/secrets.sh
export BEAKER_SESSION_SECRET=bb6624cd581d6d5192939cc1c69a020aea4168f133fff34c1b
export APP_INSTANCE_UUID=027f9157-05e4-11ea-892d-0242ac11000b
export SQLALCHEMY_URL=postgresql://first:73021fa192850023c05cbc86@db.ckan-cloud/first
export CKAN_DATASTORE_WRITE_URL=postgresql://first-datastore:c8993c5fb732c61dc82ea8ab@db.ckan-cloud/first-datastore
export CKAN_DATASTORE_READ_URL=postgresql://first-datastore-readonly:40a17bd285127229c65b2e56@db.ckan-cloud/first-datastore
export SOLR_URL=http://solr.ckan-cloud:8983/solr/first
export CKAN_REDIS_URL=redis://redis:6379/1

So as it seems everything is done in this script https://github.com/ViderumGlobal/ckan-cloud-docker/blob/master/cca-operator/cca-operator.sh. That script is executed when deployment happens https://github.com/ViderumGlobal/ckan-cloud-helm/blob/master/ckan/templates/ckan-deployment.yaml#L25-L30

Eg passwords set on line 9, 10, 11

POSTGRES_PASSWORD=`python -c "import binascii,os;print(binascii.hexlify(os.urandom(12)))"`
DATASTORE_POSTGRES_PASSWORD=`python -c "import binascii,os;print(binascii.hexlify(os.urandom(12)))"`
DATASTORE_RO_PASSWORD=`python -c "import binascii,os;print(binascii.hexlify(os.urandom(12)))"`

For some reasons we are not getting in this if statment https://github.com/ViderumGlobal/ckan-cloud-docker/blob/master/cca-operator/cca-operator.sh#L20-L38 that is responsible for creating database. Need to debug there...

Activity

  1. zelima
    self-assigned this
    on Nov 21, 2019

  2. akariv commented on Nov 21, 2019

    @akariv
    akariv
    on Nov 21, 2019 via email
    Contributor

  3. zelima commented on Nov 21, 2019

    @zelima
    zelima
    on Nov 21, 2019
    ContributorAuthor

    Which ckan image are you using?

    I'm not using any particular image right now, assuming that will build vanilla CKAN in that case https://github.com/ViderumGlobal/ckan-cloud-helm/blob/master/ckan/templates/ckan-deployment.yaml#L118. But maybe I should?

    command I've deployed the instance is folowing

    cca_helm_upgrade --install --set replicas=1 --set nginxReplicas=1 --set disableJobs=true --set useCentralizedInfra=true --set noProbes=true --set ckanOperatorImage=viderum/ckan-cloud-docker:cca-operator-latest

    Who is running cca_operator? ckan-cloud-operator?

    Not sure about that one, but my guess is yes, I see the commands are executed from this part https://github.com/ViderumGlobal/ckan-cloud-helm/blob/master/ckan/templates/ckan-deployment.yaml#L25-L30

  4. akariv commented on Nov 21, 2019

    @akariv
    akariv
    on Nov 21, 2019
    Contributor

    so - Values.ckanImage is defined to what?
    and you should be using cco to install stuff in the cluster - cca_operator is obsolete and should be deprecated asap.

  5. zelima commented on Nov 21, 2019

    @zelima
    zelima
    on Nov 21, 2019
    ContributorAuthor

    so - Values.ckanImage is defined to what?

    To None I guess.

    and you should be using cco to install stuff in the cluster - cca_operator is obsolete and should be deprecated asap.

    Can I read how to do helm based deployments via CCO anywhere? Or should I just go through the code?

  6. akariv commented on Nov 21, 2019

    @akariv
    akariv
    on Nov 21, 2019
    Contributor

    I'm not sure you can in the docs - but see this script for an example:

    https://github.com/datopian/ckan-cloud-operator/blob/master/scripts/create%20helm%20instance.py

    The same functionality is also available via the cli
    (ckan-cloud-operator instance create / ckan-dloud-operator instance update)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions