Wala shell

Advanced PHP File Manager

Server IP: 127.0.0.1 | PHP Version: 8.3.30 | Domains Hosted: 54

Name	Size	Last Modified	Actions
📁 ..	-	2025-06-25 08:26
📄 __init__.py # -- coding: utf-8 -- # Copyright 2012 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Package marker file.""" from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals	797 B	2025-06-25 08:19
📄 acls.py # -- coding: utf-8 -- # Copyright 2012 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about Access Control Lists.""" # TODO(iam-beta): Revise this to include new IAM syntax. from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>OVERVIEW</B> Access Control Lists (ACLs) allow you to control who can read and write your data, and who can read and write the ACLs themselves. If not specified at the time an object is uploaded (e.g., via the gsutil cp -a option), objects will be created with a default object ACL set on the bucket (see "gsutil help defacl"). You can replace the ACL on an object or bucket using the "gsutil acl set" command, or modify the existing ACL using the "gsutil acl ch" command (see "gsutil help acl"). <B>BUCKET VS OBJECT ACLS</B> In Google Cloud Storage, the bucket ACL works as follows: - Users granted READ access are allowed to list the bucket contents and read bucket metadata other than its ACL. - Users granted WRITE access are allowed READ access and also are allowed to write and delete objects in that bucket, including overwriting previously written objects. - Users granted OWNER access are allowed WRITE access and also are allowed to read and write the bucket's ACL. The object ACL works as follows: - Users granted READ access are allowed to read the object's data and metadata. - Users granted OWNER access are allowed READ access and also are allowed to read and write the object's ACL. A couple of points are worth noting, that sometimes surprise users: 1. There is no WRITE access for objects; attempting to set an ACL with WRITE permission for an object will result in an error. 2. The bucket ACL plays no role in determining who can read objects; only the object ACL matters for that purpose. This is different from how things work in Linux file systems, where both the file and directory permission control file read access. It also means, for example, that someone with OWNER over the bucket may not have read access to objects in the bucket. This is by design, and supports useful cases. For example, you might want to set up bucket ownership so that a small group of administrators have OWNER on the bucket (with the ability to delete data to control storage costs), but not grant those users read access to the object data (which might be sensitive data that should only be accessed by a different specific group of users). <B>ACCESSING PUBLIC OBJECTS</B> Objects with public READ access can be accessed anonymously by gsutil, via a browser, or via Cloud Storage APIs. For more details on accessing public objects, see: https://cloud.google.com/storage/docs/access-public-data <B>CANNED ACLS</B> The simplest way to set an ACL on a bucket or object is using a "canned ACL". The available canned ACLs are: project-private Gives permission to the project team based on their roles. Anyone who is part of the team has READ permission, and project owners and project editors have OWNER permission. This is the default ACL for newly created buckets. This is also the default ACL for newly created objects unless the default object ACL for that bucket has been changed. For more details see "gsutil help projects". private Gives the requester (and only the requester) OWNER permission for a bucket or object. public-read Gives all users (whether logged in or anonymous) READ permission. When you apply this to an object, anyone on the Internet can read the object without authenticating. NOTE: By default, publicly readable objects are served with a Cache-Control header allowing such objects to be cached for 3600 seconds. If you need to ensure that updates become visible immediately, you should set a Cache-Control header of "Cache-Control:private, max-age=0, no-transform" on such objects. For help doing this, see 'gsutil help setmeta'. NOTE: Setting a bucket ACL to public-read will remove all OWNER and WRITE permissions from everyone except the project owner group. Setting an object ACL to public-read will remove all OWNER and WRITE permissions from everyone except the object owner. For this reason, we recommend using the "acl ch" command to make these changes; see "gsutil help acl ch" for details. public-read-write Gives all users READ and WRITE permission. This ACL applies only to buckets. NOTE: Setting a bucket to public-read-write will allow anyone on the Internet to upload anything to your bucket. You will be responsible for this content. NOTE: Setting a bucket ACL to public-read-write will remove all OWNER permissions from everyone except the project owner group. Setting an object ACL to public-read-write will remove all OWNER permissions from everyone except the object owner. For this reason, we recommend using the "acl ch" command to make these changes; see "gsutil help acl ch" for details. authenticated-read Gives the requester OWNER permission and gives all authenticated Google account holders READ permission. bucket-owner-read Gives the requester OWNER permission and gives the bucket owner READ permission. This is used only with objects. bucket-owner-full-control Gives the requester OWNER permission and gives the bucket owner OWNER permission. This is used only with objects. <B>ACL JSON</B> When you use a canned ACL, it is translated into an JSON representation that can later be retrieved and edited to specify more fine-grained detail about who can read and write buckets and objects. By running the "gsutil acl get" command you can retrieve the ACL JSON, and edit it to customize the permissions. As an example, if you create an object in a bucket that has no default object ACL set and then retrieve the ACL on the object, it will look something like this: [ { "entity": "group-00b4903a9740e42c29800f53bd5a9a62a2f96eb3f64a4313a115df3f3a776bf7", "entityId": "00b4903a9740e42c29800f53bd5a9a62a2f96eb3f64a4313a115df3f3a776bf7", "role": "OWNER" }, { "entity": "group-00b4903a977fd817e9da167bc81306489181a110456bb635f466d71cf90a0d51", "entityId": "00b4903a977fd817e9da167bc81306489181a110456bb635f466d71cf90a0d51", "role": "OWNER" }, { "entity": "00b4903a974898cc8fc309f2f2835308ba3d3df1b889d3fc7e33e187d52d8e71", "entityId": "00b4903a974898cc8fc309f2f2835308ba3d3df1b889d3fc7e33e187d52d8e71", "role": "READER" } ] The ACL consists collection of elements, each of which specifies an Entity and a Role. Entities are the way you specify an individual or group of individuals, and Roles specify what access they're permitted. This particular ACL grants OWNER to two groups (which means members of those groups are allowed to read the object and read and write the ACL), and READ permission to a third group. The project groups are (in order) the project owners group, editors group, and viewers group. The 64 digit hex identifiers (following any prefixes like "group-") used in this ACL are called canonical IDs. They are used to identify predefined groups associated with the project that owns the bucket: the Project Owners, Project Editors, and All Project Team Members groups. For more information the permissions and roles of these project groups, see "gsutil help projects". Here's an example of an ACL specified using the group-by-email and group-by-domain entities: [ { "entity": "group-travel-companion-owners@googlegroups.com" "email": "travel-companion-owners@googlegroups.com", "role": "OWNER", } { "domain": "example.com", "entity": "domain-example.com" "role": "READER", }, ] This ACL grants members of an email group OWNER, and grants READ access to any user in a domain (which must be a Google Apps for Business domain). By applying email group grants to a collection of objects you can edit access control for large numbers of objects at once via http://groups.google.com. That way, for example, you can easily and quickly change access to a group of company objects when employees join and leave your company (i.e., without having to individually change ACLs across potentially millions of objects). <B>SHARING SCENARIOS</B> For more detailed examples how to achieve various useful sharing use cases see https://cloud.google.com/storage/docs/collaboration """) class CommandOptions(HelpProvider): """Additional help about Access Control Lists.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='acls', help_name_aliases=[ 'ACL', 'access control', 'access control list', 'authorization', 'canned', 'canned acl', ], help_type='additional_help', help_one_line_summary='Working With Access Control Lists', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	9.64 KB	2025-06-25 08:19
📄 command_opts.py # -- coding: utf-8 -- # Copyright 2012 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about gsutil command-level options.""" from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>DESCRIPTION</B> gsutil supports separate options for the top-level gsutil command and the individual sub-commands (like cp, rm, etc.) The top-level options control behavior of gsutil that apply across commands. For example, in the command: gsutil -m cp -p file gs://bucket/obj the -m option applies to gsutil, while the -p option applies to the cp sub-command. <B>OPTIONS</B> -D Shows HTTP requests/headers and additional debug info needed when posting support requests, including exception stack traces. CAUTION: The output from using this flag includes authentication credentials. Before including this flag in your command, be sure you understand how the command's output is used, and, if necessary, remove or redact sensitive information. -DD Same as -D, plus HTTP upstream payload. -h Allows you to specify certain HTTP headers, for example: gsutil -h "Cache-Control:public,max-age=3600" \\ -h "Content-Type:text/html" cp ... Note that you need to quote the headers/values that contain spaces (such as "Content-Disposition: attachment; filename=filename.ext"), to avoid having the shell split them into separate arguments. The following headers are stored as object metadata and used in future requests on the object: Cache-Control Content-Disposition Content-Encoding Content-Language Content-Type The following headers are used to check data integrity: Content-MD5 gsutil also supports custom metadata headers with a matching Cloud Storage Provider prefix, such as: x-goog-meta- Note that for gs:// URLs, the Cache Control header is specific to the API being used. The XML API accepts any cache control headers and returns them during object downloads. The JSON API respects only the public, private, no-cache, max-age, and no-transform cache control headers. See "gsutil help setmeta" for the ability to set metadata fields on objects after they have been uploaded. -i Allows you to use the configured credentials to impersonate a service account, for example: gsutil -i "service-account@google.com" ls gs://pub Note that this setting will be ignored by the XML API and S3. See 'gsutil help creds' for more information on impersonating service accounts. -m Causes supported operations (acl ch, acl set, cp, mv, rm, rsync, and setmeta) to run in parallel. This can significantly improve performance if you are performing operations on a large number of files over a reasonably fast network connection. gsutil performs the specified operation using a combination of multi-threading and multi-processing. The number of threads and processors are determined by ``parallel_thread_count`` and ``parallel_process_count``, respectively. These values are set in the .boto configuration file or specified in individual requests with the ``-o`` top-level flag. Because gsutil has no built-in support for throttling requests, you should experiment with these values. The optimal values can vary based on a number of factors, including network speed, number of CPUs, and available memory. Using the -m option can consume a significant amount of network bandwidth and cause problems or make your performance worse if you use a slower network. For example, if you start a large rsync operation over a network link that's also used by a number of other important jobs, there could be degraded performance in those jobs. Similarly, the -m option can make your performance worse, especially for cases that perform all operations locally, because it can "thrash" your local disk. To prevent such issues, reduce the values for ``parallel_thread_count`` and ``parallel_process_count``, or stop using the -m option entirely. One tool that you can use to limit how much I/O capacity gsutil consumes and prevent it from monopolizing your local disk is `ionice <http://www.tutorialspoint.com/unix_commands/ionice.htm>`_ (built in to many Linux systems). For example, the following command reduces the I/O priority of gsutil so it doesn't monopolize your local disk: ionice -c 2 -n 7 gsutil -m rsync -r ./dir gs://some bucket If a download or upload operation using parallel transfer fails before the entire transfer is complete (e.g. failing after 300 of 1000 files have been transferred), you must restart the entire transfer. Also, although most commands normally fail upon encountering an error when the -m flag is disabled, all commands continue to try all operations when -m is enabled with multiple threads or processes, and the number of failed operations (if any) are reported as an exception at the end of the command's execution. -o Override values in the `boto configuration file <https://cloud.google.com/storage/docs/boto-gsutil>`_ for the current command, in the format ``<section>:<name>=<value>``. For example, ``gsutil -o "GSUtil:parallel_thread_count=4" ...``. This does not pass the option to gsutil integration tests and does not change the values that are saved in the boto configuration file. -q Causes gsutil to perform operations quietly, i.e., without reporting progress indicators of files being copied or removed, etc. Errors are still reported. This option can be useful for running gsutil from a cron job that logs its output to a file, for which the only information desired in the log is failures. -u Allows you to specify the ID or number of a user project to be billed for the request. For example: gsutil -u "bill-this-project" cp ... """) class CommandOptions(HelpProvider): """Additional help about gsutil command-level options.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='options', help_name_aliases=['arg', 'args', 'cli', 'opt', 'opts'], help_type='additional_help', help_one_line_summary='Global Command Line Options', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	7.88 KB	2025-06-25 08:19
📄 crc32c.py # -- coding: utf-8 -- # Copyright 2013 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about CRC32C and installing crcmod.""" from __future__ import absolute_import from __future__ import division from __future__ import print_function from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>OVERVIEW</B> Google Cloud Storage provides a cyclic redundancy check (CRC) header that allows clients to verify the integrity of object contents. For non-composite objects Google Cloud Storage also provides an MD5 header to allow clients to verify object integrity, but for composite objects only the CRC is available. gsutil automatically performs integrity checks on all uploads and downloads. Additionally, you can use the ``gsutil hash`` command to calculate a CRC for any local file. The CRC variant used by Google Cloud Storage is called CRC32C (Castagnoli), which is not available in the standard Python distribution. The implementation of CRC32C used by gsutil is provided by a third-party Python module called `crcmod <https://pypi.python.org/pypi/crcmod>`_. The crcmod module contains a pure-Python implementation of CRC32C, but using it results in slow checksum computation and subsequently very poor performance. A Python C extension is also provided by crcmod, which requires compiling into a binary module for use. gsutil ships with a precompiled crcmod C extension for macOS; for other platforms, see the installation instructions below. At the end of each copy operation, the ``gsutil cp``, ``gsutil mv``, and ``gsutil rsync`` commands validate that the checksum of the source file/object matches the checksum of the destination file/object. If the checksums do not match, gsutil will delete the invalid copy and print a warning message. This very rarely happens, but if it does, you should retry the operation. <B>CONFIGURATION</B> To determine if the compiled version of crcmod is available in your Python environment, you can inspect the output of the ``gsutil version`` command for the "compiled crcmod" entry: $ gsutil version -l ... compiled crcmod: True ... If your crcmod library is compiled to a native binary, this value will be True. If using the pure-Python version, the value will be False. To control gsutil's behavior in response to crcmod's status, you can set the ``check_hashes`` variable in your `boto configuration file <https://cloud.google.com/storage/docs/boto-gsutil>`_. For details on this variable, see the surrounding comments in your boto configuration file. If ``check_hashes`` is not present in your configuration file, regenerate the file by running ``gsutil config`` with the appropriate ``-e`` or ``-a`` flag. <B>INSTALLATION</B> These installation instructions assume that: - You have ``pip`` installed. Consult the `pip installation instructions <https://pip.pypa.io/en/stable/installing/>`_ for details on how to install ``pip``. - Your installation of ``pip`` can be found in your ``PATH`` environment variable. If it cannot, you may need to replace ``pip3`` in the commands below with the full path to the executable. - You are installing the crcmod package for use with your system installation of Python, and thus use the ``sudo`` command. If installing crcmod for a different Python environment (e.g. in a virtualenv), you should omit ``sudo`` from the commands below. - You are using a Python 3 version with gsutil. You can determine which Python version gsutil is using by running ``gsutil version -l`` and looking for the ``python version: 2.x.x`` or ``python version: 3.x.x`` line. CentOS, RHEL, and Fedora ------------------------ To compile and install crcmod: yum install gcc python3-devel python3-setuptools redhat-rpm-config sudo pip3 uninstall crcmod sudo pip3 install --no-cache-dir -U crcmod Debian and Ubuntu ----------------- To compile and install crcmod: sudo apt-get install gcc python3-dev python3-setuptools sudo pip3 uninstall crcmod sudo pip3 install --no-cache-dir -U crcmod Enterprise SUSE ----------------- To compile and install crcmod when using Enterprise SUSE for SAP 12: sudo zypper install gcc python-devel sudo pip uninstall crcmod sudo pip install --no-cache-dir -U crcmod To compile and install crcmod when using Enterprise SUSE for SAP 15: sudo zypper install gcc python3-devel sudo pip uninstall crcmod sudo pip install --no-cache-dir -U crcmod macOS ----- gsutil distributes a pre-compiled version of crcmod for macOS, so you shouldn't need to compile and install it yourself. If for some reason the pre-compiled version is not being detected, please let the Google Cloud Storage team know (see ``gsutil help support``). To compile manually on macOS, you will first need to install `Xcode <https://developer.apple.com/xcode/>`_ and then run: pip3 install -U crcmod Windows ------- An installer is available for the compiled version of crcmod from the Python Package Index (PyPi) at the following URL: https://pypi.python.org/pypi/crcmod/1.7 NOTE: If you have installed crcmod and gsutil hasn't detected it, it may have been installed to the wrong directory. It should be located at <python_dir>\\files\\Lib\\site-packages\\crcmod\\ In some cases the installer will incorrectly install to <python_dir>\\Lib\\site-packages\\crcmod\\ Manually copying the crcmod directory to the correct location should resolve the issue. """) class CommandOptions(HelpProvider): """Additional help about CRC32C and installing crcmod.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='crc32c', help_name_aliases=['crc32', 'crc', 'crcmod'], help_type='additional_help', help_one_line_summary='CRC32C and Installing crcmod', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	6.5 KB	2025-06-25 08:19
📄 creds.py # -- coding: utf-8 -- # Copyright 2013 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about types of credentials and authentication.""" from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>OVERVIEW</B> gsutil currently supports several types of credentials/authentication, as well as the ability to `access public data anonymously <https://cloud.google.com/storage/docs/access-public-data>`_. Each of these type of credentials is discussed in more detail below, along with information about configuring and using credentials via the Cloud SDK. <B>Configuring/Using Credentials via Cloud SDK Distribution of gsutil</B> When gsutil is installed/used via the Cloud SDK ("gcloud"), credentials are stored by Cloud SDK in a non-user-editable file located under ~/.config/gcloud (any manipulation of credentials should be done via the gcloud auth command). If you need to set up multiple credentials (e.g., one for an individual user account and a second for a service account), the gcloud auth command manages the credentials for you, and you switch between credentials using the gcloud auth command as well (for more details, see https://cloud.google.com/sdk/gcloud/reference/auth). Once credentials have been configured via ``gcloud auth``, those credentials are used regardless of whether the user has any boto configuration files (which are located at ~/.boto unless a different path is specified in the BOTO_CONFIG environment variable). However, gsutil still looks for credentials in the boto config file if a type of non-GCS credential is needed that's not stored in the gcloud credential store (e.g., an HMAC credential for an S3 account). <B>SUPPORTED CREDENTIAL TYPES</B> gsutil supports several types of credentials (the specific subset depends on which distribution of gsutil you are using; see above discussion). OAuth2 user account: This type of credential can be used for authenticating requests on behalf of a specific user (which is probably the most common use of gsutil). This is the default type of credential that is created when you run ``gcloud init``. This credential type is not supported for stand-alone versions of gsutil. For more details about OAuth2 authentication, see: https://developers.google.com/accounts/docs/OAuth2#scenarios HMAC: This type of credential can be used by programs that are implemented using HMAC authentication, which is an authentication mechanism supported by certain other cloud storage service providers. This type of credential can also be used for interactive use when moving data to/from service providers that support HMAC credentials. This is the type of credential that is created when you run ``gsutil config -a``. Note that it's possible to set up HMAC credentials for both Google Cloud Storage and another service provider; or to set up OAuth2 user account credentials for Google Cloud Storage and HMAC credentials for another service provider. To do so, after you run the ``gcloud init`` command, you can edit the generated ~/.boto config file and look for comments for where other credentials can be added. For more details about HMAC authentication, see `HMAC keys for Cloud Storage <https://cloud.google.com/storage/docs/authentication/hmackeys>`_. OAuth2 service account: This is a credential you can use when authenticating on behalf of a service or application (as opposed to a user). For example, if you intend to run gsutil out of a nightly cron job to upload/download data, using a service account means the cron job does not depend on credentials of an individual employee at your company. This is the type of credential that is configured when you run ``gcloud auth login`` with the ``--cred-file`` flag (or ``gsutil config -e`` when using stand-alone versions of gsutil). You should generally avoid this credential type because it requires storing highly-privileged credentials in your local environment, creating potential security risks. Instead, service account impersonation or workload identity federation are the recommended alternatives when authenticating on behalf of a service or application. It is important to note that a service account is considered an Editor by default for the purposes of API access, rather than an Owner. In particular, the fact that Editors have OWNER access in the default object and bucket ACLs, but the predefined ACL options remove OWNER access from Editors, can lead to unexpected results. The solution to this problem is to use "gsutil acl ch" instead of "gsutil acl set <predefined-ACL>" to change permissions on a bucket. To set up a service account for use with gsutil, see `Authorize a service account using a service account key <https://cloud.google.com/sdk/docs/authorizing#key>`_. For more details about OAuth2 service accounts, see `Using OAuth 2.0 for server to server applications <https://developers.google.com/identity/protocols/oauth2/service-account>`_. Compute Engine internal service account: This is the type of service account used for accounts hosted by App Engine or Compute Engine. Such credentials are created automatically for you on Compute Engine when you run the ``gcloud compute instances create`` command and the credentials can be controlled with the ``--scopes`` flag. For more details about using service account credentials for authenticating workloads on Compute Engine, see https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances. For more details about App Engine service accounts, see https://developers.google.com/appengine/docs/python/appidentity/overview Service account impersonation: Impersonating a service account is useful in scenarios where you need to grant short-term access to specific resources. For example, if you have a bucket of sensitive data that is typically read-only and want to temporarily grant write access through a trusted service account. You can specify which service account to use for impersonation by running ``gsutil -i``, ``gsutil config -e`` and editing the boto configuration file, or ``gcloud config set auth/impersonate_service_account [service_account_email_address]``. In order to impersonate, your original credentials need to be granted roles/iam.serviceAccountTokenCreator on the target service account. For more information see https://cloud.google.com/docs/authentication/use-service-account-impersonation. External account credentials (workload identity federation): Using workload identity federation, you can access Google Cloud resources from Amazon Web Services (AWS), Microsoft Azure or any identity provider that supports OpenID Connect (OIDC) or SAML 2.0. For more information see https://cloud.google.com/iam/docs/workload-identity-federation. """) class CommandOptions(HelpProvider): """Additional help about types of credentials and authentication.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='creds', help_name_aliases=['credentials', 'authentication', 'auth', 'gcloud'], help_type='additional_help', help_one_line_summary='Credential Types Supporting Various Use Cases', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	8.08 KB	2025-06-25 08:19
📄 dev.py # -- coding: utf-8 -- # Copyright 2012 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about contributing code to gsutil.""" from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>OVERVIEW</B> We're open to incorporating gsutil code changes authored by users. Here are some guidelines: 1. Before we can accept code submissions, we have to jump a couple of legal hurdles. Please fill out either the individual or corporate Contributor License Agreement: - If you are an individual writing original source code and you're sure you own the intellectual property, then you'll need to sign an individual CLA (https://cla.developers.google.com/about/google-individual). - If you work for a company that wants to allow you to contribute your work to gsutil, then you'll need to sign a corporate CLA (https://cla.developers.google.com/about/google-corporate) Follow either of the two links above to access the appropriate CLA and instructions for how to sign and return it. Once we receive it, we'll add you to the official list of contributors and be able to accept your patches. 2. If you found a bug or have an idea for a feature enhancement, we suggest you check https://github.com/GoogleCloudPlatform/gsutil/issues to see if it has already been reported by another user. From there you can also subscribe to updates to the issue. If a GitHub issue doesn't already exist, create one about your idea before sending actual code. Often we can discuss the idea and help propose things that could save you later revision work. 3. We tend to avoid adding command line options that are of use to only a very small fraction of users, especially if there's some other way to accommodate such needs. Adding such options complicates the code and also adds overhead to users having to read through an "alphabet soup" list of option documentation. 4. While gsutil has a number of features specific to Google Cloud Storage, it can also be used with other cloud storage providers. We're open to including changes for making gsutil support features specific to other providers, as long as those changes don't make gsutil work worse for Google Cloud Storage. If you do make such changes we recommend including someone with knowledge of the specific provider as a code reviewer (see below). 5. You can check out the gsutil code from the GitHub repository: https://github.com/GoogleCloudPlatform/gsutil To clone a read-only copy of the repository: git clone git://github.com/GoogleCloudPlatform/gsutil.git To push your own changes to GitHub, click the Fork button on the repository page and clone the repository from your own fork. 6. The gsutil git repository uses git submodules to pull in external modules. After checking out the repository, make sure to also pull the submodules by entering into the gsutil top-level directory and run: git submodule update --init --recursive 7. Please make sure to run all tests against your modified code. To do this, change directories into the gsutil top-level directory and run: ./gsutil test The above tests take a long time to run because they send many requests to the production service. The gsutil test command has a -u argument that will only run unit tests. These run quickly, as they are executed with an in-memory mock storage service implementation. To run only the unit tests, run: ./gsutil test -u If you made changes to boto, please run the boto tests. For these tests you need to use HMAC credentials (from gsutil config -a), because the boto test suite doesn't import the OAuth2 handler. You'll also need to install some Python modules. Change directories into the boto root directory at third_party/boto and run: pip install -r requirements.txt (You probably need to run this command using sudo.) Make sure each of the individual installations succeeded. If they don't you may need to run the install command again. Then ensure your .boto file has HMAC credentials defined, and then change directories into boto's tests directory and run: python test.py unit python test.py -t s3 -t gs -t ssl 8. Please consider contributing test code for your change, especially if the change impacts any of the core gsutil code (like the gsutil cp command). 9. Please run the yapf linter with the config files in the root of the GitHub repository: yapf -irp . 10. When it's time to send us code, please submit a PR to the `gsutil GitHub repository <https://github.com/GoogleCloudPlatform/gsutil>`_. For help on making GitHub PRs, please refer to this `GitHub help document <https://help.github.com/en/articles/about-pull-requests>`_. """) class CommandOptions(HelpProvider): """Additional help about contributing code to gsutil.""" # TODO: gsutil-beta: Add lint .rc file and linting instructions. # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='dev', help_name_aliases=[ 'development', 'developer', 'code', 'mods', 'software', ], help_type='additional_help', help_one_line_summary='Contributing Code to gsutil', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	6.13 KB	2025-06-25 08:19
📄 encoding.py # -- coding: utf-8 -- # Copyright 2014 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about CRC32C and installing crcmod.""" from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>OVERVIEW</B> To reduce the chance for `filename encoding interoperability problems <https://en.wikipedia.org/wiki/Filename#Encoding_indication_interoperability>`_ gsutil uses `UTF-8 <https://en.wikipedia.org/wiki/UTF-8>`_ character encoding when uploading and downloading files. Because UTF-8 is in widespread (and growing) use, for most users nothing needs to be done to use UTF-8. Users with files stored in other encodings (such as `Latin 1 <https://en.wikipedia.org/wiki/ISO/IEC_8859-1>`_) must convert those filenames to UTF-8 before attempting to upload the files. The most common place where users who have filenames that use some other encoding encounter a gsutil error is while uploading files using the recursive (-R) option on the gsutil cp , mv, or rsync commands. When this happens you'll get an error like this: CommandException: Invalid Unicode path encountered ('dir1/dir2/file_name_with_\\xf6n_bad_chars'). gsutil cannot proceed with such files present. Please remove or rename this file and try again. Note that the invalid Unicode characters have been hex-encoded in this error message because otherwise trying to print them would result in another error. If you encounter such an error you can either remove the problematic file(s) or try to rename them and re-run the command. If you have a modest number of such files the simplest thing to do is to think of a different name for the file and manually rename the file (using local filesystem tools). If you have too many files for that to be practical, you can use a bulk rename tool or script. Unicode errors for valid Unicode filepaths can be caused by lack of Python locale configuration on Linux and Mac OSes. If your file paths are Unicode and you get encoding errors, ensure the LANG environment variable is set correctly. Typically, the LANG variable should be set to something like "en_US.UTF-8" or "de_DE.UTF-8". Note also that there's no restriction on the character encoding used in file content - it can be UTF-8, a different encoding, or non-character data (like audio or video content). The gsutil UTF-8 character encoding requirement applies only to filenames. <B>USING UNICODE FILENAMES ON WINDOWS</B> Windows support for Unicode in the command shell (cmd.exe or powershell) is somewhat painful, because Windows uses a Windows-specific character encoding called `cp1252 <https://en.wikipedia.org/wiki/Windows-1252>`_. To use Unicode characters you need to run this command in the command shell before the first time you use gsutil in that shell: chcp 65001 If you neglect to do this before using gsutil, the progress messages while uploading files with Unicode names or listing buckets with Unicode object names will look garbled (i.e., with different glyphs than you expect in the output). If you simply run the chcp command and re-run the gsutil command, the output should no longer look garbled. gsutil attempts to translate between cp1252 encoding and UTF-8 in the main places that Unicode encoding/decoding problems have been encountered to date (traversing the local file system while uploading files, and printing Unicode names while listing buckets). However, because gsutil must perform translation, it is likely there are other erroneous edge cases when using Windows with Unicode. If you encounter problems, you might consider instead using cygwin (on Windows) or Linux or macOS - all of which support Unicode. <B>USING UNICODE FILENAMES ON MACOS</B> macOS stores filenames in decomposed form (also known as `NFD normalization <https://en.wikipedia.org/wiki/Unicode_equivalence>`_). For example, if a filename contains an accented "e" character, that character will be converted to an "e" followed by an accent before being saved to the filesystem. As a consequence, it's possible to have different name strings for files uploaded from an operating system that doesn't enforce decomposed form (like Ubuntu) from one that does (like macOS). The following example shows how this behavior could lead to unexpected results. Say you create a file with non-ASCII characters on Ubuntu. Ubuntu stores that filename in its composed form. When you upload the file to the cloud, it is stored as named. But if you use gsutil rysnc to bring the file to a macOS machine and edit the file, then when you use gsutil rsync to bring this version back to the cloud, you end up with two different objects, instead of replacing the original. This is because macOS converted the filename to a decomposed form, and Cloud Storage sees this as a different object name. <B>CROSS-PLATFORM ENCODING PROBLEMS OF WHICH TO BE AWARE</B> Using UTF-8 for all object names and filenames will ensure that gsutil doesn't encounter character encoding errors while operating on the files. Unfortunately, it's still possible that files uploaded / downloaded this way can have interoperability problems, for a number of reasons unrelated to gsutil. For example: - Windows filenames are case-insensitive, while Google Cloud Storage, Linux, and macOS are not. Thus, for example, if you have two filenames on Linux differing only in case and upload both to Google Cloud Storage and then subsequently download them to Windows, you will end up with just one file whose contents came from the last of these files to be written to the filesystem. - macOS performs character encoding decomposition based on tables stored in the OS, and the tables change between Unicode versions. Thus the encoding used by an external library may not match that performed by the OS. It is possible that two object names may translate to a single local filename. - Windows console support for Unicode is difficult to use correctly. For a more thorough list of such issues see `this presentation <http://www.i18nguy.com/unicode/filename-issues-iuc33.pdf>`_ These problems mostly arise when sharing data across platforms (e.g., uploading data from a Windows machine to Google Cloud Storage, and then downloading from Google Cloud Storage to a machine running macOS). Unfortunately these problems are a consequence of the lack of a filename encoding standard, and users need to be aware of the kinds of problems that can arise when copying filenames across platforms. There is one precaution users can exercise to prevent some of these problems: When using the Windows console specify wildcards or folders (using the -R option) rather than explicitly named individual files. <B>CONVERTING FILENAMES TO UNICODE</B> Open-source tools are available to convert filenames for non-Unicode files. For example, to convert from latin1 (a common Windows encoding) to Unicode, you can use `Windows iconv <http://gnuwin32.sourceforge.net/packages/libiconv.htm>`_. For Unix-based systems, you can use `libiconv <https://www.gnu.org/software/libiconv/>`_. """) class CommandOptions(HelpProvider): """Additional help about filename encoding and interoperability problems.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='encoding', help_name_aliases=[ 'encodings', 'utf8', 'utf-8', 'latin1', 'unicode', 'interoperability', ], help_type='additional_help', help_one_line_summary='Filename encoding and interoperability problems', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	8.33 KB	2025-06-25 08:19
📄 metadata.py # -- coding: utf-8 -- # Copyright 2012 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about object metadata.""" from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>OVERVIEW OF METADATA</B> Objects can have associated metadata, which control aspects of how GET requests are handled, including ``Content-Type``, ``Cache-Control``, ``Content-Disposition``, and ``Content-Encoding``. In addition, you can set custom ``key:value`` metadata for use by your applications. For a discussion of specific metadata properties, see the `metadata concept page <https://cloud.google.com/storage/docs/metadata>`_. There are two ways to set metadata on objects: - At upload time you can specify one or more metadata properties to associate with objects, using the ``gsutil -h option``. For example, the following command would cause gsutil to set the ``Content-Type`` and ``Cache-Control`` for each of the files being uploaded from a local directory named ``images``: gsutil -h "Content-Type:text/html" \\ -h "Cache-Control:public, max-age=3600" cp -r images \\ gs://bucket/images Note that -h is an option on the gsutil command, not the cp sub-command. - You can set or remove metadata fields from already uploaded objects using the ``gsutil setmeta`` command. See "gsutil help setmeta". <B>SETTABLE FIELDS; FIELD VALUES</B> You can't set some metadata fields, such as ETag and Content-Length. The fields you can set are: - ``Cache-Control`` - ``Content-Disposition`` - ``Content-Encoding`` - ``Content-Language`` - ``Content-Type`` - ``Custom-Time`` - Custom metadata Field names are case-insensitive. All fields and their values must consist only of ASCII characters, with the exception of values for ``x-goog-meta-`` fields, which may contain arbitrary Unicode values. Note that when setting metadata using the XML API, which sends custom metadata as HTTP headers, Unicode characters are encoded using UTF-8, then url-encoded to ASCII. For example: gsutil setmeta -h "x-goog-meta-foo: ã" gs://bucket/object stores the custom metadata key-value pair of ``foo`` and ``%C3%A3``. Subsequently, running ``ls -L`` using the JSON API to list the object's metadata prints ``%C3%A3``, while ``ls -L`` using the XML API url-decodes this value automatically, printing the character ``ã``. <B>VIEWING CURRENTLY SET METADATA</B> You can see what metadata is currently set on an object by using: gsutil ls -L gs://the_bucket/the_object """) class CommandOptions(HelpProvider): """Additional help about object metadata.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='metadata', help_name_aliases=[ 'cache-control', 'caching', 'content type', 'mime type', 'mime', 'type', ], help_type='additional_help', help_one_line_summary='Working With Object Metadata', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	3.74 KB	2025-06-25 08:19
📄 naming.py # -- coding: utf-8 -- # Copyright 2012 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about gsutil object and bucket naming.""" from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>BUCKET NAME REQUIREMENTS</B> Google Cloud Storage has a single namespace, so you will not be allowed to create a bucket with a name already in use by another user. You can, however, carve out parts of the bucket name space corresponding to your company's domain name (see "DOMAIN NAMED BUCKETS"). Bucket names must conform to standard DNS naming conventions. This is because a bucket name can appear in a DNS record as part of a CNAME redirect. In addition to meeting DNS naming requirements, Google Cloud Storage imposes other requirements on bucket naming. At a minimum, your bucket names must meet the following requirements: - Bucket names must contain only lowercase letters, numbers, dashes (-), and dots (.). - Bucket names must start and end with a number or letter. - Bucket names must contain 3 to 63 characters. Names containing dots can contain up to 222 characters, but each dot-separated component can be no longer than 63 characters. - Bucket names cannot be represented as an IPv4 address in dotted-decimal notation (for example, 192.168.5.4). - Bucket names cannot begin with the "goog" prefix. - For DNS compliance, you should not have a period adjacent to another period or dash. For example, ".." or "-." or ".-" are not acceptable. <B>OBJECT NAME REQUIREMENTS</B> Object names can contain any sequence of Unicode characters, of length 1-1024 bytes when UTF-8 encoded. Object names must not contain CarriageReturn, CarriageReturnLineFeed, or the XML-disallowed surrogate blocks (xFFFE or xFFFF). We strongly recommend that you abide by the following object naming conventions: - Avoid using control characters that are illegal in XML 1.0 in your object names (#x7F-#x84 and #x86-#x9F). These characters will cause XML listing issues when you try to list your objects. - Avoid using "#" in your object names. gsutil interprets object names ending with #<numeric string> as version identifiers, so including "#" in object names can make it difficult or impossible to perform various operations on such objects using gsutil (see 'gsutil help versions'). - Avoid using "[", "]", "*", or "?" in your object names. gsutil interprets these characters as wildcards, so including any of these characters in object names can make it difficult or impossible to perform various wildcard operations using gsutil (see 'gsutil help wildcards'). See also 'gsutil help encoding' about file/object name encoding requirements and potential interoperability concerns. <B>DOMAIN NAMED BUCKETS</B> You can carve out parts of the Google Cloud Storage bucket name space by creating buckets with domain names (like "example.com"). Before you can create a bucket name containing one or more '.' characters, the following rules apply: - If the name is a syntactically valid DNS name ending with a currently-recognized top-level domain (such as .com), you will be required to verify domain ownership. - Otherwise you will be disallowed from creating the bucket. If your project needs to use a domain-named bucket, you need to have a team member both verify the domain and create the bucket. This is because Google Cloud Storage checks for domain ownership against the user who creates the bucket, so the user who creates the bucket must also be verified as an owner or manager of the domain. To verify as the owner or manager of a domain, use the Google Webmaster Tools verification process. The Webmaster Tools verification process provides three methods for verifying an owner or manager of a domain: 1. Adding a special Meta tag to a site's homepage. 2. Uploading a special HTML file to a site. 3. Adding a DNS TXT record to a domain's DNS configuration. Meta tag verification and HTML file verification are easier to perform and are probably adequate for most situations. DNS TXT record verification is a domain-based verification method that is useful in situations where a site wants to tightly control who can create domain-named buckets. Once a site creates a DNS TXT record to verify ownership of a domain, it takes precedence over meta tag and HTML file verification. For example, you might have two IT staff members who are responsible for managing your site, called "example.com." If they complete the DNS TXT record verification, only they would be able to create buckets called "example.com", "reports.example.com", "downloads.example.com", and other domain-named buckets. Site-Based Verification ----------------------- If you have administrative control over the HTML files that make up a site, you can use one of the site-based verification methods to verify that you control or own a site. When you do this, Google Cloud Storage lets you create buckets representing the verified site and any sub-sites - provided nobody has used the DNS TXT record method to verify domain ownership of a parent of the site. As an example, assume that nobody has used the DNS TXT record method to verify ownership of the following domains: abc.def.example.com, def.example.com, and example.com. In this case, Google Cloud Storage lets you create a bucket named abc.def.example.com if you verify that you own or control any of the following sites: http://abc.def.example.com http://def.example.com http://example.com Domain-Based Verification ------------------------- If you have administrative control over a domain's DNS configuration, you can use the DNS TXT record verification method to verify that you own or control a domain. When you use the domain-based verification method to verify that you own or control a domain, Google Cloud Storage lets you create buckets that represent any subdomain under the verified domain. Furthermore, Google Cloud Storage prevents anybody else from creating buckets under that domain unless you add their name to the list of verified domain owners or they have verified their domain ownership by using the DNS TXT record verification method. For example, if you use the DNS TXT record verification method to verify your ownership of the domain example.com, Google Cloud Storage will let you create bucket names that represent any subdomain under the example.com domain, such as abc.def.example.com, example.com/music/jazz, or abc.example.com/music/jazz. Using the DNS TXT record method to verify domain ownership supersedes verification by site-based verification methods. For example, if you use the Meta tag method or HTML file method to verify domain ownership of http://example.com, but someone else uses the DNS TXT record method to verify ownership of the example.com domain, Google Cloud Storage will not allow you to create a bucket named example.com. To create the bucket example.com, the domain owner who used the DNS TXT method to verify domain ownership must add you to the list of verified domain owners for example.com. The DNS TXT record verification method is particularly useful if you manage a domain for a large organization that has numerous subdomains because it lets you control who can create buckets representing those domain names. Note: If you use the DNS TXT record verification method to verify ownership of a domain, you cannot create a CNAME record for that domain. RFC 1034 disallows inclusion of any other resource records if there is a CNAME resource record present. If you want to create a CNAME resource record for a domain, you must use the Meta tag verification method or the HTML file verification method. """) class CommandOptions(HelpProvider): """Additional help about gsutil object and bucket naming.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='naming', help_name_aliases=['domain', 'limits', 'name', 'names'], help_type='additional_help', help_one_line_summary='Object and Bucket Naming', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	8.83 KB	2025-06-25 08:19
📄 prod.py # -- coding: utf-8 -- # Copyright 2012 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about using gsutil for production tasks.""" from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals from gslib.help_provider import HelpProvider from gslib.utils.constants import RESUMABLE_THRESHOLD_MIB _DETAILED_HELP_TEXT = (""" <B>OVERVIEW</B> If you use gsutil in large production tasks (such as uploading or downloading many GiBs of data each night), there are a number of things you can do to help ensure success. Specifically, this section discusses how to script large production tasks around gsutil's resumable transfer mechanism. <B>BACKGROUND ON RESUMABLE TRANSFERS</B> First, it's helpful to understand gsutil's resumable transfer mechanism, and how your script needs to be implemented around this mechanism to work reliably. gsutil uses resumable transfer support when you attempt to download a file of any size or to upload a file larger than a configurable threshold (by default, this threshold is %d MiB). If a transfer fails partway through (e.g., because of an intermittent network problem), gsutil uses a `truncated randomized binary exponential backoff-and-retry strategy <https://cloud.google.com/storage/docs/retry-strategy#tools>`_ that by default retries transfers up to 23 times over a 10 minute period of time. If the transfer fails each of these attempts with no intervening progress, gsutil gives up on the transfer, but keeps a "tracker" file for it in a configurable location (the default location is ~/.gsutil/, in a file named by a combination of the SHA1 hash of the name of the bucket and object being transferred and the last 16 characters of the file name). When transfers fail in this fashion, you can rerun gsutil at some later time (e.g., after the networking problem has been resolved), and the resumable transfer picks up where it left off. <B>SCRIPTING DATA TRANSFER TASKS</B> To script large production data transfer tasks around this mechanism, you can implement a script that runs periodically, determines which file transfers have not yet succeeded, and runs gsutil to copy them. Below, we offer a number of suggestions about how this type of scripting should be implemented: 1. When resumable transfers fail without any progress 23 times in a row over the course of up to 10 minutes, it probably won't work to simply retry the transfer immediately. A more successful strategy would be to have a cron job that runs every 30 minutes, determines which transfers need to be run, and runs them. If the network experiences intermittent problems, the script picks up where it left off and will eventually succeed (once the network problem has been resolved). 2. If your business depends on timely data transfer, you should consider implementing some network monitoring. For example, you can implement a task that attempts a small download every few minutes and raises an alert if the attempt fails for several attempts in a row (or more or less frequently depending on your requirements), so that your IT staff can investigate problems promptly. As usual with monitoring implementations, you should experiment with the alerting thresholds, to avoid false positive alerts that cause your staff to begin ignoring the alerts. 3. There are a variety of ways you can determine what files remain to be transferred. We recommend that you avoid attempting to get a complete listing of a bucket containing many objects (e.g., tens of thousands or more). One strategy is to structure your object names in a way that represents your transfer process, and use gsutil prefix wildcards to request partial bucket listings. For example, if your periodic process involves downloading the current day's objects, you could name objects using a year-month-day-object-ID format and then find today's objects by using a command like gsutil ls "gs://bucket/2011-09-27-". Note that it is more efficient to have a non-wildcard prefix like this than to use something like gsutil ls "gs://bucket/-2011-09-27". The latter command actually requests a complete bucket listing and then filters in gsutil, while the former asks Google Storage to return the subset of objects whose names start with everything up to the "*". For data uploads, another technique would be to move local files from a "to be processed" area to a "done" area as your script successfully copies files to the cloud. You can do this in parallel batches by using a command like: gsutil -m cp -r to_upload/subdir_$i gs://bucket/subdir_$i where i is a shell loop variable. Make sure to check the shell $status variable is 0 after each gsutil cp command, to detect if some of the copies failed, and rerun the affected copies. With this strategy, the file system keeps track of all remaining work to be done. 4. If you have really large numbers of objects in a single bucket (say hundreds of thousands or more), you should consider tracking your objects in a database instead of using bucket listings to enumerate the objects. For example this database could track the state of your downloads, so you can determine what objects need to be downloaded by your periodic download script by querying the database locally instead of performing a bucket listing. 5. Make sure you don't delete partially downloaded temporary files after a transfer fails: gsutil picks up where it left off (and performs a hash of the final downloaded content to ensure data integrity), so deleting partially transferred files will cause you to lose progress and make more wasteful use of your network. 6. If you have a fast network connection, you can speed up the transfer of large numbers of files by using the gsutil -m (multi-threading / multi-processing) option. Be aware, however, that gsutil doesn't attempt to keep track of which files were downloaded successfully in cases where some files failed to download. For example, if you use multi-threaded transfers to download 100 files and 3 failed to download, it is up to your scripting process to determine which transfers didn't succeed, and retry them. A periodic check-and-run approach like outlined earlier would handle this case. If you use parallel transfers (gsutil -m) you might want to experiment with the number of threads being used (via the parallel_thread_count setting in the .boto config file). By default, gsutil uses 10 threads for Linux and 24 threads for other operating systems. Depending on your network speed, available memory, CPU load, and other conditions, this may or may not be optimal. Try experimenting with higher or lower numbers of threads to find the best number of threads for your environment. """ % RESUMABLE_THRESHOLD_MIB) class CommandOptions(HelpProvider): """Additional help about using gsutil for production tasks.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='prod', help_name_aliases=[ 'production', 'resumable', 'resumable upload', 'resumable transfer', 'resumable download', 'scripts', 'scripting', ], help_type='additional_help', help_one_line_summary='Scripting Production Transfers', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	8.12 KB	2025-06-25 08:19
📄 security.py # -- coding: utf-8 -- # Copyright 2015 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about types of credentials and authentication.""" from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>OVERVIEW</B> This help section provides details about various precautions taken by gsutil to protect data security, as well as recommendations for how customers should safeguard security. <B>TRANSPORT LAYER SECURITY</B> gsutil performs all operations using transport-layer encryption (HTTPS), to protect against data leakage over shared network links. This is also important because gsutil uses "bearer tokens" for authentication (OAuth2) as well as for resumable upload identifiers, and such tokens must be protected from being eavesdropped and reused. gsutil also supports the older HMAC style of authentication via the XML API (see `gsutil endpoints <https://cloud.google.com/storage/docs/request-endpoints#gsutil>`_). While HMAC authentication does not use bearer tokens (and thus is not subject to eavesdropping/replay attacks), it's still important to encrypt data traffic. To add an extra layer of security, gsutil supports mutual TLS (mTLS) for the Cloud Storage JSON API. With mTLS, the client verifies the server certificate, and the server also verifies the client. To find out more about how to enable mTLS, see the `install docs <https://cloud.google.com/storage/docs/gsutil_install>`_. <B>LOCAL FILE STORAGE SECURITY</B> gsutil takes a number of precautions to protect against security exploits in the files it stores locally: - When the ``gcloud init``, ``gsutil config -a``, or ``gsutil config -e`` commands run, they set file protection mode 600 ("-rw-------") on the .boto configuration file they generate, so only the user (or superuser) can read it. This is important because these files contain security-sensitive information, including credentials and proxy configuration. - These commands also use file protection mode 600 for the private key file stored locally when you create service account credentials. - The default level of logging output from gsutil commands does not include security-sensitive information, such as OAuth2 tokens and proxy configuration information. (See the "RECOMMENDED USER PRECAUTIONS" section below if you increase the level of debug output, using the gsutil -D option.) Note that protection modes are not supported on Windows, so if you use gsutil on Windows we recommend using an encrypted file system and strong account passwords. <B>SECURITY-SENSITIVE FILES WRITTEN TEMPORARILY TO DISK BY GSUTIL</B> gsutil buffers data in temporary files in several situations: - While compressing data being uploaded via gsutil cp -z/-Z, gsutil buffers the data in temporary files with protection 600, which it deletes after the upload is complete (similarly for downloading files that were uploaded with gsutil cp -z/-Z or some other process that sets the Content-Encoding to "gzip"). However, if you kill the gsutil process while the upload is under way the partially written file will be left in place. See the "CHANGING TEMP DIRECTORIES" section in "gsutil help cp" for details of where the temporary files are written and how to change the temp directory location. - When performing a resumable upload gsutil stores the upload ID (which, as noted above, is a bearer token and thus should be safe-guarded) in a file under ~/.gsutil/tracker-files with protection 600, and deletes this file after the upload is complete. However, if the upload doesn't complete successfully the tracker file is left in place so the resumable upload can be re-attempted later. Over time it's possible to accumulate these tracker files from aborted upload attempts, though resumable upload IDs are only valid for 1 week, so the security risk only exists for files less than that age. If you consider the risk of leaving aborted upload IDs in the tracker directory too high you could modify your upload scripts to delete the tracker files; or you could create a cron job to clear the tracker directory periodically. - The gsutil rsync command stores temporary files (with protection 600) containing the names, sizes, and checksums of source and destination directories/buckets, which it deletes after the rsync is complete. However, if you kill the gsutil process while the rsync is under way the listing files will be left in place. Note that gsutil deletes temporary files using the standard OS unlink system call, which does not perform `data wiping <https://en.wikipedia.org/wiki/Data_erasure>`_. Thus, the content of such temporary files can be recovered by a determined adversary. <B>ACCESS CONTROL LISTS</B> Unless you specify a different ACL (e.g., via the gsutil cp -a option), by default objects written to a bucket use the default object ACL on that bucket. Unless you modify that ACL (e.g., via the gsutil defacl command), by default it will allow all project editors write access to the object and read/write access to the object's metadata and will allow all project viewers read access to the object. The Google Cloud Storage access control system includes the ability to specify that objects are publicly readable. Make sure you intend for any objects you write with this permission to be public. Once "published", data on the Internet can be copied to many places, so it's effectively impossible to regain read control over an object written with this permission. The Google Cloud Storage access control system includes the ability to specify that buckets are publicly writable. While configuring a bucket this way can be convenient for various purposes, we recommend against using this permission - it can be abused for distributing illegal content, viruses, and other malware, and the bucket owner is legally and financially responsible for the content stored in their buckets. If you need to make content available to customers who don't have Google accounts consider instead using signed URLs (see "gsutil help signurl"). <B>SOFTWARE INTEGRITY AND UPDATES</B> gsutil is distributed as a part of the bundled Cloud SDK release. This distribution method takes a variety of security precautions to protect the integrity of the software. We strongly recommend against getting a copy of gsutil from any other sources (such as mirror sites). <B>PROXY USAGE</B> gsutil supports access via proxies, such as Squid and a number of commercial products. A full description of their capabilities is beyond the scope of this documentation, but proxies can be configured to support many security-related functions, including virus scanning, Data Leakage Prevention, control over which certificates/CA's are trusted, content type filtering, and many more capabilities. Some of these features can slow or block legitimate gsutil behavior. For example, virus scanning depends on decrypting file content, which in turn requires that the proxy terminate the gsutil connection and establish a new connection - and in some cases proxies will rewrite content in ways that result in checksum validation errors and other problems. For details on configuring proxies, see the proxy help text generated in your .boto configuration file by the ``gcloud init``, ``gsutil -a``, and ``gsutil -e`` commands. <B>ENCRYPTION AT REST</B> All Google Cloud Storage data are automatically stored in an encrypted state, but you can also provide your own encryption keys. For more information, see `Cloud Storage Encryption <https://cloud.google.com/storage/docs/encryption>`_. <B>DATA PRIVACY</B> Google will never ask you to share your credentials, password, or other security-sensitive information. Beware of potential phishing scams where someone attempts to impersonate Google and asks for such information. <B>MEASUREMENT DATA</B> The gsutil perfdiag command collects a variety of performance-related measurements and details about your local system and network environment, for use in troubleshooting performance problems. None of this information will be sent to Google unless you choose to send it. <B>RECOMMENDED USER PRECAUTIONS</B> The first and foremost precaution is: Never share your credentials. Each user should have distinct credentials. If you run gsutil -D (to generate debugging output) it will include OAuth2 refresh and access tokens in the output. Make sure to redact this information before sending this debug output to anyone during troubleshooting/tech support interactions. If you run gsutil --trace-token (to send a trace directly to Google), sensitive information like OAuth2 tokens and the contents of any files accessed during the trace may be included in the content of the trace. Customer-supplied encryption key information in the .boto configuration is security sensitive. The proxy configuration information in the .boto configuration is security-sensitive, especially if your proxy setup requires user and password information. Even if your proxy setup doesn't require user and password, the host and port number for your proxy is often considered security-sensitive. Protect access to your .boto configuration file. If you are using gsutil from a production environment (e.g., via a cron job running on a host in your data center), use service account credentials rather than individual user account credentials. These credentials were designed for such use and, for example, protect you from losing access when an employee leaves your company. """) class CommandOptions(HelpProvider): """Additional help about security and privacy considerations using gsutil.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='security', help_name_aliases=['protection', 'privacy', 'proxies', 'proxy'], help_type='additional_help', help_one_line_summary='Security and Privacy Considerations', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	10.72 KB	2025-06-25 08:19
📄 shim.py # -- coding: utf-8 -- # Copyright 2013 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about the shim for running gcloud storage.""" from __future__ import absolute_import from __future__ import division from __future__ import print_function from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>OVERVIEW</B> Cloud SDK includes a new CLI, gcloud storage, that can be considerably faster than gsutil when performing uploads and downloads with less parameter tweaking. This new CLI has a syntax and command structure that is familiar to gsutil users but is fundamentally different in many important ways. To ease transition to this new CLI, gsutil provides a shim that translates your gsutil commands to gcloud storage commands if an equivalent exists, and falls back to gsutil's usual behavior if an equivalent does not exist. <B> PREREQUISITES </B> You should be running gsutil binary installed via Cloud SDK tool. To verify this, run ``gsutil ver -l`` and you should be able to see ``pass cloud sdk credentials to gsutil: True``. If it is ``False``, try running ``gcloud config set pass_credentials_to_gsutil True``. If you do not have ``gcloud`` installed, follow the `Cloud SDK installation instructions. <https://cloud.google.com/sdk/docs/install>`_. <B>TO ENABLE</B> Set ``use_gcloud_storage=True`` in the ``.boto`` config file under the ``[GSUtil]`` section: [GSUtil] use_gcloud_storage=True You can also set the flag for individual commands using the top-level ``-o`` flag: gsutil -o "GSUtil:use_gcloud_storage=True" -m cp -p file gs://bucket/obj <B>AVAILABLE COMMANDS</B> The gcloud storage CLI only supports a subset of gsutil commands. What follows is a list of commands supported by the shim with any differences in behavior noted. acl ------------------------ - The ``ch`` subcommand is not supported. autoclass ------------------------ - Works as expected. bucketpolicyonly ------------------------ - Works as expected. cat ------------------------ - Prints object data for a second object even if the first object is invalid. compose ------------------------ - Works as expected. cors ------------------------ - ``get`` subcommand prints "[]" instead of "gs://[bucket name] has no CORS configuration". cp ------------------------ - Copies a second object even if the first object is invalid. - Does not support file to file copies. - Supports copying objects cloud-to-cloud with trailing slashes in the name. - The all-version flag (``-A``) silently enables sequential execution rather than raising an error. defacl ------------------------ - The ``ch`` subcommand is not supported. defstorageclass ------------------------ - Works as expected. hash ------------------------ - In gsutil, the ``-m`` and ``-c`` flags that affect which hashes are displayed are ignored for cloud objects. This behavior is fixed for the shim and gcloud storage. iam ------------------------ - The ``ch`` subcommand is not supported. - The ``-f`` flag will continue on any error, not just API errors. kms ------------------------ - The authorize subcommand returns informational messages in a different format. - The encryption subcommand returns informational messages in a different format. labels ------------------------ - ``get`` subcommand prints "[]" instead of "gs://[bucket name] has no labels configuration." lifecycle ------------------------ - Works as expected. logging ------------------------ - The get subcommand has different JSON spacing and doesn't print an informational message if no configuration is found. ls ------------------------ - `ls -L` output uses spaces instead of tabs. mb ------------------------ - Works as expected. mv ------------------------ - See notes on cp. notification ------------------------ - The list subcommand prints configuration information as YAML. - The delete subcommand offers progress tracking and parallelization. pap ------------------------ - Works as expected. rb ------------------------ - Works as expected. requesterpays ------------------------ - Works as expected. rewrite ------------------------ - The -k flag does not throw an error if called without a new key. In both the shim and unshimmed cases, the old key is maintained. rm ------------------------ - ``$folder$`` delete markers are not supported. rpo ------------------------ - Works as expected. setmeta ------------------------ - Does not throw an error if no headers are changed. stat ------------------------ - Includes a field "Storage class update time:" which may throw off tabbing. ubla ------------------------ - Works as expected. versioning ------------------------ - Works as expected. web ------------------------ - The get subcommand has different JSON spacing and doesn't print an informational message if no configuration is found. <B>BOTO CONFIGURATION</B> Configuration found in the boto file is mapped 1:1 to gcloud environment variables where appropriate. Important: The ``gs_oauth2_refresh_token`` and ``gs_service_key_file`` fields are ignored by the shim and authentication is taken care by ``gcloud``. [Credentials] ------------------------ - aws_access_key_id: AWS_ACCESS_KEY_ID - aws_secret_access_key: AWS_SECRET_ACCESS_KEY - use_client_certificate: CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE [Boto] ------------------------ - proxy: CLOUDSDK_PROXY_ADDRESS - proxy_type: CLOUDSDK_PROXY_TYPE - proxy_port: CLOUDSDK_PROXY_PORT - proxy_user: CLOUDSDK_PROXY_USERNAME - proxy_pass: CLOUDSDK_PROXY_PASSWORD - proxy_rdns: CLOUDSDK_PROXY_RDNS - http_socket_timeout: CLOUDSDK_CORE_HTTP_TIMEOUT - ca_certificates_file: CLOUDSDK_CORE_CUSTOM_CA_CERTS_FILE - max_retry_delay: CLOUDSDK_STORAGE_BASE_RETRY_DELAY - num_retries: CLOUDSDK_STORAGE_MAX_RETRIES [GSUtil] ------------------------ - check_hashes: CLOUDSDK_STORAGE_CHECK_HASHES - default_project_id: CLOUDSDK_CORE_PROJECT - disable_analytics_prompt: CLOUDSDK_CORE_DISABLE_USAGE_REPORTING - use_magicfile: CLOUDSDK_STORAGE_USE_MAGICFILE - parallel_composite_upload_threshold: CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_THRESHOLD - resumable_threshold: CLOUDSDK_STORAGE_RESUMABLE_THRESHOLD [OAuth2] ------------------------ - client_id: CLOUDSDK_AUTH_CLIENT_ID - client_secret: CLOUDSDK_AUTH_CLIENT_SECRET - provider_authorization_uri: CLOUDSDK_AUTH_AUTH_HOST - provider_token_uri: CLOUDSDK_AUTH_TOKEN_HOST <B>GENERAL COMPATIBILITY NOTES</B> - Due to its compatibility across all major platforms, multiprocessing is enabled for all commands by default (equivalent to the -m option always being included in gsutil). - A sequence of asterisks greater than 2 (i.e. ``**``) are always treated as a single asterisk. - Unlike gsutil, gcloud is not designed to be used in parallel invocations, and doing so (i.e. running the shim from 2 terminals at once) can lead to unpredictable behavior. - Assuming a bucket contains an object ``gs://bucket/nested/foo.txt``, gsutil's wildcard iterator will match ``foo.txt`` given a URL like ``gs://bucket//nested/*``. The shim will not match ``foo.txt`` given the same URL. - This will be updated as new commands are supported by both gcloud storage and the shim. - If Unicode is having issues, try setting the environment variable ``PYTHONUTF8`` to ``1``. Specifically, this may help on the Windows command-line (CMD). """) class CommandOptions(HelpProvider): """Additional help about the shim for running gcloud storage.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='shim', help_name_aliases=['gcloudstorage', 'gcloud storage'], help_type='additional_help', help_one_line_summary='Shim for Running gcloud storage', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	8.64 KB	2025-06-25 08:19
📄 support.py # -- coding: utf-8 -- # Copyright 2012 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about technical and billing support.""" from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>TECHNICAL SUPPORT</B> If you have any questions or encounter any problems with Google Cloud Storage, please first read the `FAQ <https://cloud.google.com/storage/docs/faq>`_. If you still have questions please use one of the following methods as appropriate, providing the details noted below: A) For API, tool usage, or other software development-related questions, please search for and post questions on Stack Overflow, using the official `google-cloud-storage tag <http://stackoverflow.com/questions/tagged/google-cloud-storage>`_. Our support team actively monitors questions to this tag and we'll do our best to respond. B) For gsutil bugs or feature requests, please check if there is already a `existing GitHub issue <https://github.com/GoogleCloudPlatform/gsutil/issues>`_ that covers your request. If not, create a `new GitHub issue <https://github.com/GoogleCloudPlatform/gsutil/issues/new>`_. To help us diagnose any issues you encounter, when creating a new issue please provide these details in addition to the description of your problem: - The resource you are attempting to access (bucket name, object name), assuming they are not sensitive. - The operation you attempted (GET, PUT, etc.) - The time and date (including timezone) at which you encountered the problem - If you can use gsutil to reproduce your issue, specify the -D option to display your request's HTTP details, and provide these details in the issue. Warning: The gsutil -d, -D, and -DD options will also print the authentication header with authentication credentials for your Google Cloud Storage account. Make sure to remove any "Authorization:" headers before you post HTTP details to the issue. Note also that if you upload files large enough to use resumable uploads, the resumable upload IDs are security-sensitive while an upload is not yet complete, so should not be posted on public forums. If you make any local modifications to gsutil, please make sure to use a released copy of gsutil (instead of your locally modified copy) when providing the gsutil -D output noted above. We cannot support versions of gsutil that include local modifications. (However, we're open to user contributions; see "gsutil help dev".) <B>BILLING AND ACCOUNT QUESTIONS</B> A) For billing documentation, please visit https://cloud.google.com/storage/pricing. If you want to cancel billing, follow the instructions at `Cloud Storage FAQ <https://cloud.google.com/storage/docs/faq#disablebilling>`_. Caution: When you disable billing, you also disable the Google Cloud Storage service. Make sure you want to disable the Google Cloud Storage service before you disable billing. B) For support regarding billing, please see `billing support <https://support.google.com/cloud/contact/cloud_platform_billing>`_. For other questions regarding your account, Terms Of Service, Google Cloud Console, or other administration-related questions please see `Google Cloud Platform support <https://support.google.com/cloud/answer/6282346#gcp>`_. """) class CommandOptions(HelpProvider): """Additional help about technical and billing support.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='support', help_name_aliases=[ 'techsupport', 'tech support', 'technical support', 'billing', 'faq', 'questions', ], help_type='additional_help', help_one_line_summary='Google Cloud Storage Support', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	4.49 KB	2025-06-25 08:19
📄 versions.py # -- coding: utf-8 -- # Copyright 2012 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Additional help about object versioning.""" from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals from gslib.help_provider import HelpProvider _DETAILED_HELP_TEXT = (""" <B>OVERVIEW</B> Versioning-enabled buckets maintain noncurrent versions of objects, providing a way to un-delete data that you accidentally deleted, or to retrieve older versions of your data. Noncurrent objects are ignored by gsutil commands unless you indicate it should do otherwise by setting a relevant command flag or by including a specific generation number in your command. For example, wildcards like ```` and ``*`` do not, by themselves, act on noncurrent object versions. When using gsutil cp, you cannot specify a version-specific URL as the destination, because writes to Cloud Storage always create a new version. Trying to specify a version-specific URL as the destination of ``gsutil cp`` results in an error. When you specify a noncurrent object as a source in a copy command, you always create a new object version and retain the original (even when using the command to restore a live version). You can use the ``gsutil mv`` command to simultaneously restore an object version and remove the noncurrent copy that was used as the source. You can turn versioning on or off for a bucket at any time. Turning versioning off leaves existing object versions in place and simply causes the bucket to delete the existing live version of the object whenever a new version is uploaded. Regardless of whether you have enabled versioning on a bucket, every object has two associated positive integer fields: - the generation, which is updated when a new object replaces an existing object with the same name. Note that there is no guarantee that generation numbers increase for successive versions, only that each new version has a unique generation number. - the metageneration, which identifies the metadata generation. It starts at 1; is updated every time the metadata (e.g., ACL or Content-Type) for a given content generation is updated; and gets reset when the generation number changes. Of these two integers, only the generation is used when working with versioned data. Both generation and metageneration can be used with concurrency control. To learn more about versioning and concurrency, see the following documentation: - `Overview of Object Versioning <https://cloud.google.com/storage/docs/object-versioning>`_ - `Guide for using Object Versioning <https://cloud.google.com/storage/docs/using-object-versioning>`_ - The `reference page for the gsutil versioning command <https://cloud.google.com/storage/docs/gsutil/commands/versioning>`_ - `Request preconditions <https://cloud.google.com/storage/docs/request-preconditions>`_ """) class CommandOptions(HelpProvider): """Additional help about object versioning.""" # Help specification. See help_provider.py for documentation. help_spec = HelpProvider.HelpSpec( help_name='versions', help_name_aliases=['concurrency', 'concurrency control'], help_type='additional_help', help_one_line_summary='Object Versioning and Concurrency Control', help_text=_DETAILED_HELP_TEXT, subcommand_help_text={}, )	3.93 KB	2025-06-25 08:19

Terminal