| Server IP : 198.38.94.13 / Your IP : 216.73.217.33 Web Server : Apache System : Linux d4744.dxb1.stableserver.net 5.14.0-611.49.1.el9_7.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 21 16:39:08 EDT 2026 x86_64 User : revivere ( 1140) PHP Version : 8.2.33 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : OFF | Pkexec : OFF Directory : /opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/__pycache__/ |
Upload File : |
o
;j�= �
@ s0 d Z ddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl Z ddl Z ddl
Z
ddlZddlZddlZddlZddlZddlZddlmZ ddlmZ ddlZddlZddlZddlZddl Zddl Zddl Zddl Zddl Zddl!Zddl"Zddl#Zddl$Zddl%Zddl&Zddl'Zddl(m)Z) ddl(m*Z+ ddl,m-Z-m.Z.m/Z/ ddl0m1Z1 dd l"m2Z3 dd
l'm4Z4 e+d Z5d Z6d
Z7dZ8dZ9dZ:dZ;dZ<dZ=dZ>e �?e@�ZAdZBzddlCZCdZBW n eDy� Y nw z.ejEjF�G� �rejEjH�Id�dk�rddlJmKZK eK� �reDd��ddlLZLddlMZMe4eLjN�ZOW n eP�y+ dZOY nw ztddlQZQeQ�R� � eQ�Sd� d e jTv�rJe jH�Ud �e jTd < ddlVZVW d � n 1 �sYw Y e4eVjN�ZWe4eVjX�ZXeWe4d �k�r�z
ddlYZVddlZZVW n
eD�y� Y n w d d � Z[e[eVj\_]zeVj^j_Z_W n e`�y� ePZ_Y nw W n" eP�y� Za zdZWdZXebeaeD��s�eA�cd!� W Y dZa[andZa[aww e4d"�Zde4d#�Zee4d$�Zfd%d&� Zgd'd(� ZhG d)d*� d*�ZiG d+d,� d,ei�ZjG d-d.� d.ei�Zkekejd/�ZlG d0d1� d1�ZmG d2d3� d3em�ZnG d4d5� d5em�ZoG d6d7� d7em�Zpd8d9� ZqdS ):zJ
Classes which provide the shared base for GitFS, git_pillar, and winrepo
� N)� OrderedDict)�datetime)�DEFAULT_HASH_TYPE)�DEFAULT_MASTER_OPTS)�FileserverConfigError� GitLockError�get_error_message)�tagify)�get_machine_identifier)�VersionZgitfs_ref_types��name� �d )�pygit2)�user�password�pubkey�privkey�
passphrase�
insecure_auth)�
mountpoint�root�refzgGitPython is installed, you may wish to set %s_provider to 'gitpython' to use GitPython for %s support.zbpygit2 is installed, you may wish to set %s_provider to 'pygit2' to use pygit2 for for %s support.z�Cache path %s (corresponding remote: %s) exists but is not a valid git repository. You will need to manually delete this directory on the master to continue to use this %s remote.FT�gitz /usr/bin/git)� git_is_stubzGit is not present.�ignore�HOME�~z0.26.3c C s | sd S t j�| ��d�S )N�utf-8)r Zffi�string�decode)�ptrr r �D/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py�__maybe_string� s r$ zFailed to import pygit2z0.3z0.20.3z0.20.0c s� t t t ddddtd�� � fdd�}| � vr!|| �} | du r!t|�S � | }|dkrKt|tftf�s5t|�}t|t�rDdd� |�d�D �S d d� |D �S z||�W S tyd t�d
| |� t|� Y S w ) zO
Force params to be strings unless they should remain a different type
Z
stringlist)�
ssl_verifyr �disable_saltenv_mapping�saltenv_whitelist�saltenv_blacklist�refspecs� ref_types�update_intervalc s` � D ])}z| � d| �r|}W |S W q ty+ | � dt|� �r)|}Y |S Y qw d }|S )N�_)�endswith� TypeError�str)�key�item�ret�Znon_string_paramsr r# � _find_global� s � ��z#enforce_types.<locals>._find_globalNc S � g | ]}|� � �qS r )�strip��.0�xr r r# �
<listcomp>� � z!enforce_types.<locals>.<listcomp>�,c S s g | ]}t |��qS r �r/ r7 r r r# r: � r; zGFailed to enforce type for key=%s with val=%s, falling back to a string) �bool�intr/ �
isinstance�list�split� Exception�log�error)r0 �valr4 �expectedr r3 r# �
enforce_types� s> �
� �rH c C s t d| � ���)z7
Fatal configuration issue, raise an exception
zFailed to load )r )�roler r r# �failhard� s rJ c @ s` e Zd ZdZe�� Z dSdd�Zdd� Zdd� Z d d
� Z
d d � Z d
d� Z dd� Z
dd� Zdd� ZdTdd�Zedd� �Zdd� Zd d � ZdTd d �ZdTd d!�Zd"d#� Zd$d%� ZdUd'd(�ZdUd)d*�Zd+d,� Ze j dVd/d0��Z d1d2� Z dWd4d5�Z d6d7� Z!d8d9� Z"d:d;� Z#d<d=� Z$d>d?� Z%d@dA� Z&dBdC� Z'dDdE� Z(dFdG� Z)dHdI� Z*e+dJdK� �Z,dLdM� Z-dNdO� Z.dPdQ� Z/dRS )X� GitProviderz�
Base class for gitfs/git_pillar provider classes. Should never be used
directly.
self.provider should be set in the sub-class' __init__ function before
invoking the parent class' __init__.
�gitfsc
s || _ || _dd� }t� �dd�| _tjjj| j �| j� d�g �ddt |d�| _
t � |�| _
| j
�� D ]\} }
d|
v rLtjj�| j
| d �| j
| d< q3� fd d
�|D �} | rat�d d �| �� z|� �W n tyx t|�t� � �Y nw t|t��r'tt|��| _| �� tjjj|| j ddt t d�} | s�t�d
| j| j� t | j� | jdkr�| j dkr�d| v r�t�d| j| j� t | j� d}
�fdd�| D �D ]G}d}
|t v r�| j t!vr�d�"| j|| jd �t!�| j �}| jdkr�|d7 }t�|� q�d�"| j|| jd ����}| jdk�r|d7 }t�|� q�|
�r t | j� | j
�#| � n|| _| �� d| j
v�r9d| j
d< | jd k�rhd | j
v�rh| j�$d d �d | j
d < | j
d �%� �&d!��rh| j
d d d"� | j
d < d| j
v �rtjj�| j
d ��'d �| j
d< nd| j
d< d#| j
v�r�i | j
d#< n$| j
d# �� D ] \} }
d|
v �r�| j
d# | }tjj�|d �|d< �q�| j
�� D ]\}}|t(v�r�t)| |��s�t*| ||� �q�t(D ]}|d$k�r�t*| d%| | j
| � | �+|� �q�t)| d&��s#| j� d'�}zt,| }W n t-�y t�d(|� t | j� Y nw t*| d&|� t�.d)|| j| j� t/| d*� t)| d+��rXd,d
� | j0D �| _0d-d
� | j0D �}|�rXt�d.| j| jd �|�d �t1�� t | j� t| jt ��smt�d/| j| j� t | j� t)| d ��rx| j2| _3n t4t5| j �d0t6��}t t7�8|| j�9d1���:� �d2d3��;d d%�| _3tjj<�|| j3�| _=d%| _>| j�?d��r�z| �@� | _>W n tA�y� t�d4| j| j� t | j� Y nw tjj<�| j3| j>�| _Btjj<�| j=| j>�| _Ctjj<�|d5| jB�| _Dtjj<�|d6| jB�| _EtFj<�G| jC��stF�H| jC� z| �I� | _JW n2 tK�y? } z%d7�"| j| j|�}t| tL��r)|d87 }tj|dd9� t | j� W Y d }~nd }~ww | �M� | �N� tFj<�G| jD��sVtF�H| jD� | �O� tP�r�tF�Q� }tR�S|�}| j �d:d �}| j �d;d �}|�r�|�r�tjjT�UtV|� d S d S d S d S )<Nc S s t |�S �Nr= )r9 �yr r r# �_val_cb s z%GitProvider.__init__.<locals>._val_cb�
machine_id�no_machine_id_availableZ_saltenvT)�strictZrecurseZkey_cbZval_cbr c s g | ]}|� v r|�qS r r r7 )�per_remote_onlyr r# r: $ � z(GitProvider.__init__.<locals>.<listcomp>ziThe following parameter names are restricted to per-remote use only: %s. This is a bug, please report it.�, z�Invalid per-remote configuration for %s remote '%s'. If no per-remote parameters are being specified, there may be a trailing colon after the URL, which should be removed. Check the master configuration file.�
git_pillar�__env__�basez{Invalid per-remote configuration for %s remote '%s'. base can only be specified if __env__ is specified as the branch name.Fc 3 s � | ] }|� vr|V qd S rM r r7 )�valid_per_remote_paramsr r# � <genexpr>V s � �z'GitProvider.__init__.<locals>.<genexpr>z�{0} authentication parameter '{1}' (from remote '{2}') is only supported by the following provider(s): {3}. Current {0}_provider is '{4}'.rL zLSee the GitFS Walkthrough in the Salt documentation for further information.zQInvalid {} configuration parameter '{}' in remote '{}'. Valid parameters are: {}.�M See the GitFS Walkthrough in the Salt documentation for further information.r � �winrepor
�/� ����.git����saltenvr r, r) Z _refspecsz@The '%s' option has no default value in salt/config/__init__.py.ztThe 'refspecs' option was not explicitly defined as a configurable parameter. Falling back to %s for %s remote '%s'.�confr* c S r5 r )�lowerr7 r r r# r: � r; c S s g | ]}|t vr|�qS r )�VALID_REF_TYPESr7 r r r# r: � rT zXThe following ref_types for %s remote '%s' are invalid: %s. The supported values are: %szZInvalid %s remote '%s'. Remotes must be strings, you may need to enclose the URL in quotes� hash_typer �ascii)�encodingz%__env__ cant generate basename: %s %s�work�linksz6Exception caught while initializing {} remote '{}': {}z Perhaps git is not available.��exc_info�cachedirZ
gitfs_remotes)W�optsrI �_get_machine_identifier�get�mach_id�salt�utils�dataZrepack_dictlistr/ �global_saltenv�copy�deepcopyrd �items�urlZ strip_protorD �critical�joinr. rA r@ �dict�next�iter�id�get_urlrH rJ �branch� AUTH_PARAMS�provider�AUTH_PROVIDERS�format�update�rsplitre r- r6 �PER_SALTENV_PARAMS�hasattr�setattr�add_conf_overlay�_DEFAULT_MASTER_OPTS�KeyError�debug�delattrr* rf r
�_cache_basehash�getattr�hashlibr �base64� b64encode�encode�digest�replace�path� _cache_hash�_cache_basename�
startswith�get_checkout_target�AttributeError�_cache_full_basename� _cachedir�_salt_working_dir�_linkdir�os�isdir�makedirs� init_remote�newrC � GitPython� verify_auth�setup_callbacks�fetch_request_check�
HAS_PSUTIL�getpid�psutilZProcess�processZ"register_cleanup_finalize_function�gitfs_finalize_cleanup)�selfro �remote�per_remote_defaultsrS �override_params�
cache_rootrI rO rc � saltenv_confZper_remote_collisionsZper_remote_confZper_remote_errors�param�msgZ saltenv_ptrr0 rF Zdefault_refspecsZinvalid_ref_typesrg �exc�cur_pidr� � cache_dirZ gitfs_activer )rS rY r# �__init__� s�
�� ��� � ��
�
���
� �� �
�
�
�� �
�� �
�
�
���
�����
� ��
��zGitProvider.__init__c C � | j S rM )r� �r� r r r# �get_cache_basehash& � z GitProvider.get_cache_basehashc C r� rM )r� r� r r r# �get_cache_hash) r� zGitProvider.get_cache_hashc C r� rM )r� r� r r r# �get_cache_basename, r� z GitProvider.get_cache_basenamec C r� rM )r� r� r r r# �get_cache_full_basename/ r� z#GitProvider.get_cache_full_basenamec C r� rM )r� r� r r r# � get_cachedir2 r� zGitProvider.get_cachedirc C r� rM )r� r� r r r# � get_linkdir5 r� zGitProvider.get_linkdirc C r� rM )r� r� r r r# �get_salt_working_dir8 r� z GitProvider.get_salt_working_dirc
s� � fdd�}d� j v }d� j v }t� }tjj�� j�r |�d� tjj� |�D ]=}|�
d�r4|dd� }|� d d
�\}}|d krY|rY|� d �} | d rO| d n| d
}|||� q'|dkrd|rd|||� q'|S )z�
Return the names of remote refs (stripped of the remote name) and tags
which are map to the branches and tags.
c sN |� j v r| �� j | � dS |� jkr| �d� dS � js%| �|� dS dS )zC
Add the appropriate saltenv(s) to the set
rX N)�saltenv_revmapr� rX �addr& )Zenv_set�rnamer� r r# �
_check_refA s
�z8GitProvider._get_envs_from_ref_paths.<locals>._check_refr� �tagrX zrefs/� Nr^ r_ �remotes� r �tags)
r* �setrs rt � stringutils�is_hexrX r� ru r! r� rB � partition)
r� �refsr� Z use_branchesZuse_tagsr2 r Zrtyper� Zpartedr r� r# �_get_envs_from_ref_paths; s$
�z$GitProvider._get_envs_from_ref_pathsr� c C s t jj�| j|d �S )N�.lk)rs rt r� r| r� �r� � lock_typer r r# �_get_lock_fileb s zGitProvider._get_lock_filec s d� fdd� }t | � |� dS )zV
Programmatically determine config value based on the desired saltenv
rX c s( �fdd�}�j dkr|t�d� ��S �j�|i �� �dkrl�� �fdd�}||�}z�j}|rB||krBt�d||�j �j||� |W S tyM Y nw |d krU�j S �j
rh|d u rft�d
�j �j|� |S |pk|S �� v rv|� � �S |�j v r���j | v r�|�j | � �S |t�d� ��S ) Nc s � dv r
| � tj�S | S )N)r r )�rstripr� �sep)r9 r r r# � strip_sepl s zAGitProvider.add_conf_overlay.<locals>._getconf.<locals>.strip_seprL r, r c s: � �v r�� S | �j v r� �j | v r�j | � S d S rM )rv )�tgt_env)r
r� r� r r# �_get_per_saltenvu s
zHGitProvider.add_conf_overlay.<locals>._getconf.<locals>._get_per_saltenvz�The per-saltenv configuration has mapped the '%s' branch/tag to saltenv '%s' for %s remote '%s', but this remote has all_saltenvs set to '%s'. The per-saltenv mapping will be ignored in favor of '%s'.rX zXsaltenv mapping is disabled for %s remote '%s' and saltenv '%s' is not explicitly mapped) rI r� rc rq � all_saltenvsrD r� r� r� rX r& rv )r� r� r� r� Zper_saltenv_refZall_saltenvs_refr )r� r� r# �_getconfk sP
�
�� z.GitProvider.add_conf_overlay.<locals>._getconfN�rX )r� )�clsr
r� r r r# r� e s Dz GitProvider.add_conf_overlayc C sJ t jj�| j| �� ��tj�}tj� |�r|S t
� d| �� | j | j
� dS )z�
Check if the relative root path exists in the checked-out copy of the
remote. Return the full path to that relative root if it does exist,
otherwise return None.
z7Root path '%s' not present in %s remote '%s', skipping.N)rs rt r� r| r� r r� r� r� r� rD rE rI r� )r� �root_dirr r r# �
check_root� s �zGitProvider.check_rootc C s� g }d}t j�� }tjj�� sd|d< d|d< tjt �
|�tjj�� t j � | j
�|tjtjd�}|�� d }|�t�}|jdkrNt�d| j| j||� |S d}tjj�
|d �D ]}|�|�rl|�|t|�d
� � � � qX|rzt� d | jd � |�� |S )
zV
Remove stale refs so that they are no longer seen as fileserver envs
zgit remote prune origin� Cs LANGUAGEs LC_ALL)� close_fds�cwd�env�stdout�stderrr zOFailed to prune stale branches for %s remote '%s'. Output from '%s' follows:
%sz * [pruned] �
Nz&%s pruned the following stale refs: %srU ) r� �environrw rs rt �platform�
is_windows�
subprocess�Popen�shlexrB r� �dirname�gitdir�PIPE�STDOUT� communicater! �__salt_system_encoding__�
returncoderD �warningrI r� � itertoolsr� �append�lenr6 r� r| )r� �cleanedZcmd_strr� �cmd�output�marker�liner r r# �clean_stale_refs� sJ
�
��
��z GitProvider.clean_stale_refsc C sN | j jjdd�du rt�d� td��z | �|�W | j j�� S | j j�� w )z!
Clear update.lk
�< ��timeoutF�gitfs master lock timeout!)� __class__� _master_lock�acquirerD rE � TimeoutError� _clear_lock�releaser� r r r# �
clear_lock� s
zGitProvider.clear_lockc
sX �j |d�� � �fdd�}g }g }zt�� � W ns ty� } zg|jtjkr9d�j� d� � d|� d�}t�|� n3|jtj kr^zt
� � � W n. ty] } z |||� W Y d}~n%d}~ww |||� W Y d}~||fS W Y d}~||fS W Y d}~||fS W Y d}~||fS d}~ww d |� d
�j � d �j
� d �j� d
� }t�|� |�|� ||fS )z?
Clear update.lk without MultiProcessing locks
�r� c s( d� �j� |�}t�|� | �|� d S )Nz-Unable to remove update lock for {} ({}): {} )r� rz rD r� r� )Zerrlistr� r� �� lock_filer� r r# �
_add_error s
�
z+GitProvider._clear_lock.<locals>._add_errorzAttempt to remove lock z for file (z$) which does not exist, exception : � NzRemoved �
lock for � remote '�' on machine_id '�')r� r� �remove�OSError�errno�ENOENTrz rD r� �EISDIR�shutil�rmtreerI r� rr r� )r� r� r �success�failedr� r� r r r# r sJ �� ��
�
�
�
����
zGitProvider._clear_lockc C sN t j�| jd�}tjj�� }|�|�st �
d|� dS d}d}z|� |d�}W n tjjj y< |�
|� d}d}Y nw t �d| j| j|| j� || jkrc|�|d| j� t �d | j| j| j� d}z t|j |d
dd ��}W n tjjjy} g }Y nw t| j�}t �d | j| j||� ||kr�|�|d
|� t �d
| j| j|� d}z|� dd�}W n tjjj y� |�
d� d}Y n tjjjy� d}Y nw t| j��� } t �d| j| j|| � || kr�|�dd| � t �d| j| j| � d}|�r%tjj�|d��}
|� |
� t �d| j| j|� W d � dS 1 �s w Y dS dS )z�
For the config options which need to be maintained in the git config,
ensure that the git config file is configured as desired.
�configz&Failed to read from git config file %sFzremote "origin"rz TNz6Current fetch URL for %s remote '%s': %s (desired: %s)z&Fetch URL for %s remote '%s' set to %s�fetch)Zas_listz5Current refspecs for %s remote '%s': %s (desired: %s)z%Refspecs for %s remote '%s' set to %s�httpZ sslVerifyz;Current http.sslVerify for %s remote '%s': %s (desired: %s)z+http.sslVerify for %s remote '%s' set to %s�wz/Config updates for %s remote '%s' written to %s) r� r� r| r� rs rt Z configparserZGitConfigParser�readrD rE rq ZNoSectionErrorZ add_sectionr� rI r� rz r� �sortedZ
NoOptionErrorr) Z set_multivarr/ r% re �files�fopen�write) r� �
git_configrd Z conf_changedZremote_sectionrz r) Zdesired_refspecsr% Zdesired_ssl_verify�fp_r r r# �enforce_git_config0 s�
��
��
��
���
�$��z GitProvider.enforce_git_configc C s� z'| j dd�� t�d| j| j� | �� W d � W S 1 s w Y W dS tyd } z1|jtjkrFt� d| j| j| j
dd�| j� nt� d| j| j| j
dd�| jt |�� W Y d}~dS d}~w t y| } z
t� d|� W Y d}~dS d}~ww ) z�
Fetch the repo. If the local copy was updated, return True. If the
local copy was already up-to-date, return False.
This function requires that a _fetch() function be implemented in a
sub-class.
r� r zFetching %s remote '%s'Na1 Update lock file is present for %s remote '%s', skipping. If this warning persists, it is possible that the update process was interrupted, but the lock could also have been manually set. Removing %s or running 'salt-run cache.clear_git_lock %s type=update' will allow updates to continue for this remote.��Update lock file generated an unexpected exception for %s remote '%s', The lock file %s for %s type=update operation, exception: %s .Fz*fetch got NotImplementedError exception %s)
�gen_lockrD r� rI r� �_fetchr r �EEXISTr� r� r/ �NotImplementedError)r� r� r r r# r! � s8 (�
�
� ���zGitProvider.fetchFc C sP | j jjdd�du rt�d� td��z
| �||�W | j j�� S | j j�� w )zO
Place a lock file if (and only if) it does not already exist.
r r Fr )r r r rD rE r
�_GitProvider__lockr )r� r� rJ r r r# �_lock� s
zGitProvider._lockc
C s� z=t �| �|�t jt jB t jB �}t �|d�� t �|tj j
� t � � � d| j
� d��� W d � n1 s6w Y W �nx t�y� } �zj|jtjk�r�tj j�| �|�d��E}zttj j
�|�� ��� �}W n tyt d}Y nw z
tj j
�|�� ��� }W n ty� } zd}W Y d}~nd}~ww W d � n1 s�w Y | jd }| j|d�} | j| �r<|� d|� d | � d
| j� d | j� d | j
� d
|� d�}
|�r,|
d|� d�7 }
| j
s�|r�|
d|� d| j
� �7 }
tj j� |��s,| j
|k�r|
d|� �7 }
n*|
d7 }
t � |
� | � � \} } | �r!| j d|d�W Y d}~S |�r%� W Y d}~dS t � |
� |�r5� W Y d}~dS |�r^tj j� |��r^t � d|| j|| | j
� |�rW� W Y d}~dS |�rmt � d|| j|| | j
� | � � \} } | �r�| j d|d�W Y d}~S |�r�� W Y d}~dS d|� d| j� d| �|�� d | j
� d |� �
}
t j!|
d d � t"|j|
��d}~ww d |� d| j� d | j� d!| j
� d"� }
t �#|
� |
S )#zv
Place a lock file if (and only if) it does not already exist.
Without MultiProcessing locks.
�wbr� N�rr Z _global_lockr z is enabled and z
lockfile z is present for r z' on machine_id z with pid 'z'.z Process z obtained the lockz for machine_id z, current machine_id z� but this process is not running. The update may have been interrupted. If using multi-master with shared gitfs cache, the lock may have been obtained by another master, with machine_id z� but this process is not running. The update may have been interrupted. Given this process is for the same machine the lock will be reallocated to new process r� �r� rJ z1Process %d has a %s %s lock (%s) on machine_id %szjProcess %d has a %s %s lock (%s) on machine_id %s, but this process is not running. Cleaning up lock file.zUnable to set r z (z) on machine_id �: Trl zSet r r )$r� �openr� �O_CREAT�O_EXCL�O_WRONLY�fdopenr( rs rt r� �to_bytesr� rr r r r/ r& r' r? �
to_unicode�readliner� �
ValueErrorrI ro r� r� Z
os_is_runningrD r� r r1 rE r r� )
r� r� rJ Zfh_r� �fd_�pidrr Zglobal_lock_keyr r� r Zfailr r r# Z__lock� s� � � ��� ������ �
�����
���
�
�� ���� ��$d
zGitProvider.__lockc C s� g }g }z| j dd�}W n. ty: } z"t�d| j| j| jdd�| jt|�� |�|j � W Y d}~||fS d}~ww |durD|�|� ||fS )aG
Place an lock file and report on the success/failure. This is an
interface to be used by the fileserver runner, so it is hard-coded to
perform an update lock. We aren't using the gen_lock()
contextmanager here because the lock is meant to stay and not be
automatically removed.
r� r r, N)
r2 r rD r� rI r� r� r/ r� �strerror)r� r r �resultr� r r r# �lockH s(
� ��
zGitProvider.lockr � �?c c s� � t |t�sttjd|� d���zt|�}W n ty! d}Y nw |dk r(d}t |tftf�r4|dk r6d}||kr<|}d}d}zst�� } z| j |dd� d}|V d}W n5 t
tfy� } z'|rjt�� | |krqt|j|j ��t �
d|| j| j|� t�|� W Y d }~qEd }~ww W |s�|r�d
|� d | j� d | j� d
|� d|� d� }t �
|� | j|d� d S d S |s�|r�d
|� d | j� d | j� d
|� d|� d� }t �
|� | j|d� w w )z�
Set and automatically clear a lock,
should be called from a context, for example: with self.gen_lock()
zInvalid lock_type 'r r rE FTr5 zFA %s lock is already present for %s remote '%s', sleeping %f second(s)NzAttempting to remove 'z ' lock for 'z
' remote 'z' due to lock_set1 'z' or lock_set2 'r )r@ r/ r r �EINVALr? r? �float�timer2 r rB rD r� rI r� �sleepr
) r� r� r Z
poll_intervalZ lock_set1Z lock_set2�
time_startr� r� r r r# r- d s� �
��
��������
������
�zGitProvider.gen_lockc C � t � ��zA
This function must be overridden in a sub-class
�r0 r� r r r# r� � � zGitProvider.init_remoteTc C rK rL rM )r� �
fetch_on_failr r r# �checkout� rN zGitProvider.checkoutc C rK rL rM �r� r� r r r# �dir_list� rN zGitProvider.dir_listc C s t jjj|| j| jd�S )zo
Check if an environment is exposed by comparing it against a whitelist
and blacklist.
)Z whitelistZ blacklist)rs rt r� Zcheck_whitelist_blacklistr' r( rQ r r r# �env_is_exposed� s
�zGitProvider.env_is_exposedc C rK )zb
Provider-specific code for fetching, must be implemented in a
sub-class.
rM r� r r r# r. � s zGitProvider._fetchc C rK rL rM r� r r r# �envs� rN zGitProvider.envsc C rK rL rM rQ r r r# � file_list� rN zGitProvider.file_listc C rK rL rM )r� r� r� r r r# � find_file� rN zGitProvider.find_filec C sh | j dkr1| jdkr1z| jW S ty Y nw | j�d�p%| j�d�p%d}|dkr-| jS t|�S | jS )z0
Resolve dynamically-set branch
rV rW � pillarenvrc rX )rI r� r� r� ro rq rX r/ )r� �targetr r r# r� � s � z GitProvider.get_checkout_targetc C s� | � |�sdS | �|�}|du rdS | jD ]-}z d|� �}t| |�}W n ty5 t�d| jj|� Y qw ||�}|durB| S q| j rx| jD ].}z d|� �}t| |�}W n tyi t�d| jj|� Y qIw || j �}|durw| S qIdS )zD
Return a tree object for the specified environment
NZget_tree_from_z!%s class is missing function '%s')
rS r r* r� r� rD rE r �__name__�fallback)r� r� �tgt_refZref_typeZ func_name�func� candidater r r# �get_tree� sJ
���
��
�zGitProvider.get_treec C s\ | j dv r(z| j�dd�\| _| _W dS ty' | jd | _| j| _Y dS w | j| _dS )zW
Examine self.id and assign self.url (and self.branch, for git_pillar)
)rV r] Nr_ r� )rI r� rB r� rz r? rd r� r r r# r� s
� zGitProvider.get_urlc
C s� t jj�| jd�}tj�|�r@t�d| j� zt� |� W n t
y9 } ztj d| j|dd� W Y d }~nd }~ww | � � dS dS )N�
fetch_requestzFetch request: %sz%Failed to remove Fetch request: %s %sTrl F)
rs rt r� r| r� r� �isfilerD r� r r rE r! )r� r_ r� r r r# r� ' s" ���z GitProvider.fetch_request_checkc C s z| j W S ty� g | _ z| j�d�}W n ty' t�d| jj� Y nVw t|dd� �D ]2\}}z ||d g}W n t yH g }Y nw | j �
t j j
j| jg|d|d � �R � |g f� q0z| j �d| j|d gg f� W n t y| Y nw | j Y S w )zq
Return the expected result of an os.walk on the linkdir, based on the
mountpoint value.
r^ z-%s class is missing a '_mountpoint' attributeNr` r_ r )Z
_linkdir_walkr� � _mountpointrB rD rE r rY � enumerate�
IndexErrorr� rs rt r� r| r� �insert)r� �parts�idxr1 �dirsr r r# � linkdir_walk8 s> �� �"�� �
�zGitProvider.linkdir_walkc C � dS )zX
Only needed in pygit2, included in the base class for simplicty of use
Nr r� r r r# r� ^ s zGitProvider.setup_callbacksc C s
d| _ dS )zS
Override this function in a sub-class to implement auth checking.
NT�� credentialsr� r r r# r� c s zGitProvider.verify_authc C rK rL rM )r� �blob�destr r r# �
write_filej rN zGitProvider.write_fileN�rL )r� )r� F)r� r rE �T)0rY �
__module__� __qualname__�__doc__�multiprocessing�Lockr r� r� r� r� r� r� r� r� r� r� � classmethodr� r� r r
r r+ r! r2 r1 rD �
contextlib�contextmanagerr- r� rP rR rS r. rT rU rV r� r^ r� r� �propertyrh r� r� rn r r r r# rK � s\
� )
'
K
0
.j
)
x A
-
% rK c s~ e Zd ZdZ d � fdd� Zd dd�Zdd � Zd
d � Zd d
� Zdd� Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd� Z� ZS ) r� z
Interface to GitPython
rL c �" d| _ t� �|||||||� d S �N� gitpython�r� �superr� �r� ro r� r� rS r� r� rI �r r r# r� v �
�zGitPython.__init__Tc C s� | � � | �� }z | j�d�j}W n ty d}Y nw d| dfd| dfg}| jr<|d| j dfd| j dfg7 }|D ]�\}}z | j�|�j}W n tyT Y q>w ||kr_| �� S z.| jdd�� | jj �
|� t � d | j
| j||rzd
nd � W d � n1 s�w Y W n< ty� } z(|jtjkr�t|jd | j
� d
| j� d���t �d|j| j
| j� W Y d}~ dS d}~w ty� Y q>w | �� S |r�t � d|| j
| j� | �� | j
dd�S t �d|| j
| j� dS )af
Checkout the configured branch/tag. We catch an "Exception" class here
instead of a specific exception class because the exceptions raised by
GitPython when running these functions vary in different versions of
GitPython.
fetch_on_fail
If checkout fails perform a fetch then try to checkout again.
�HEADNzorigin/Fztags/TrP r z+%s remote '%s' has been checked out to %s%sz as fallbackr\ �Checkout lock exists for r r �?Error %d encountered obtaining checkout lock for %s remote '%s'�>Failed to checkout %s from %s remote '%s': fetch and try again�rO �DFailed to checkout %s from %s remote '%s': remote ref does not exist)r� r� �repo� rev_parse�hexsharC rZ r� r- r rP rD r� rI r� r r r/ rE r! ) r� rO r[ �head_shaZ
checkout_refs� checkout_refrZ �
target_shar� r r r# rP � s�
�
� � �
� �� ��� � � �zGitPython.checkoutc C s� d}t �| j�stj�| j�| _d}n"z t�| j�| _W n tjjy4 t �
t | j| j | j
� | Y S w tjj�| jjd�| _| �� |S )z�
Initialize/attach to a remote using GitPython. Return a boolean
which will let the calling function know whether or not a new repo was
initialized by this function.
FTra )r� �listdirr� r ZRepo�initr� r� ZInvalidGitRepositoryErrorrD rE �
_INVALID_REPOrz rI rs rt r� r| Z working_dirr� r+ )r� r� r r r# r� � s �zGitPython.init_remotec s� t � }� ���}|s |S � ���r.z |� ��� }W n ty% | Y S w � �fdd�}ndd� }� �fdd�}|�� D ]}t|tj�rO|�|||j ��� q=� �
��r]|�� �
��� |S )zT
Get list of directories for the target environment using GitPython
c � t j�| � ����S rM �r� r� �relpathr �r� rQ r r# r�
� z#GitPython.dir_list.<locals>.relpathc S � | S rM r r� r r r# r� � c � t jjj� ���| dd�S �NT�Z
use_posixpath�rs rt r� r| r r� rQ r r# �add_mountpoint � �z*GitPython.dir_list.<locals>.add_mountpoint) r� r^ r r� �traverser@ r �Treer� r� r )r� r� r2 �treer� r� rl r rQ r# rR � s(
� �
zGitPython.dir_listc C s dd� | j jD �}| �|�S )�n
Check the refs and return a list of the ones which can be used as salt
environments.
c S s g | ]}|j �qS r r� r7 r r r# r: # s z"GitPython.envs.<locals>.<listcomp>)r� r� r� �r� Z ref_pathsr r r# rT s
zGitPython.envsc C s� | j jd }z|�� }W n
ty |�� }Y nw d}|D ]B}|jdurAt�d| j|j| j |jj
dd� |j j
dd� � d}q |j |j
|jfv r`t�d| j|j |j
krWdnd |j| j � d}q | �� }|si|rkdS dS )
��
Fetch the repo. If the local copy was updated, return True. If the
local copy was already up-to-date, return False.
r FNz1%s has updated '%s' for remote '%s' from %s to %s� Tz*%s has fetched new %s '%s' for remote '%s'r� �head)r� r� r! �AssertionErrorZ
old_commitrD r� rI r
r� r� �commit�flagsZNEW_TAGZNEW_HEADr )r� �origin�
fetch_resultsZnew_objsZ fetchinfor� r r r# r. & s< �
���zGitPython._fetchc s t � }i }� ���}|s||fS � ���r4z |� ��� }W n
ty+ ||f Y S w � �fdd�}ndd� }� �fdd�}|�� D ]:}t|tj�sLqC|||j��}|� |� t
� |j �r}t
�� } |�| � | �d� tjj�| �� �}
| �� |
||< qC||fS )zJ
Get file list for the target environment using GitPython
c r� rM r� r� rQ r r# r� \ r� z$GitPython.file_list.<locals>.relpathc S r� rM r r� r r r# r� a r� c r� r� r� r� rQ r r# r� d r� z+GitPython.file_list.<locals>.add_mountpointr )r� r^ r r� r� r@ r �Blobr� r� �stat�S_ISLNK�mode�io�BytesIO� stream_data�seekrs rt r� �to_strr$ �close) r� r� r& �symlinksr� r� r� � file_blobZ file_path�stream�link_tgtr rQ r# rU L s8
�
�zGitPython.file_listc C s� | � |�}|s dS d}d} |d7 }|tkrd}nOzB|| }t�|j�rNt�� }|�|� |�d� t j
j � |�
� �}|�� t j
jjtj�|�|dd�}n |}t|tj�rXd}W nW n tyf d}Y nw qt|tj�ru||j|jfS dS �zF
Find the specified file in the specified environment
)NNNNr Tr_ r� )r^ �SYMLINK_RECURSE_DEPTHr� r� r� r� r� r� r� rs rt r� r� r$ r� r� r| r� r� r@ r r� r� r� r� ) r� r� r� r� rl �depthr� r� r� r r r# rV w sD
� � �� zGitPython.find_filec C �0 z
t �| jd|� ��jjW S ty Y dS w )zh
Return a git.Tree object matching a head ref fetched into
refs/remotes/origin/
�refs/remotes/origin/N)r ZRemoteReferencer� r� r� r? �r� r r r r# �get_tree_from_branch� s �� �z GitPython.get_tree_from_branchc C r� )zU
Return a git.Tree object matching a tag ref fetched into refs/tags/
�
refs/tags/N)r Z TagReferencer� r� r� r? r� r r r# �get_tree_from_tag� s
�zGitPython.get_tree_from_tagc C s. z| j �|�jW S tjjtfy Y dS w )z9
Return a git.Tree object matching a SHA
N)r� r� r� �gitdbr� ZODBErrorr� r� r r r# �get_tree_from_sha� �
�zGitPython.get_tree_from_shac C s@ t jj�|d��}|�|� W d � dS 1 sw Y dS �zO
Using the blob object, write the file to the destination path
�wb+N)rs rt r& r' r� �r� rl rm r* r r r# rn � s "�zGitPython.write_filero rp )rY rq rr rs r� rP r� rR rT r. rU rV r� r� r� rn �
__classcell__r r r� r# r� q s �
Y"&+, r� c s� e Zd ZdZ d%� fdd� Zdd� Zd&dd �Zd'� fd d � Zd
d� Zdd� Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd � Zd d � Zd d � Zd!d"� Zd#d$� Z� ZS )(�Pygit2z
Interface to Pygit2
rL c rz �Nr r} r r� r r# r� � r� zPygit2.__init__c C s& z|� � W S ty |�� Y S w )aY
Compatibility function for pygit2.Reference objects. Older versions of
pygit2 use .get_object() to return the object to which the reference
points, while newer versions use .peel(). In pygit2 0.27.4,
.get_object() was removed. This function will try .peel() first and
fall back to .get_object().
)�peelr� Z
get_object)r� �objr r r# r� � s
�z Pygit2.peelTc
s� �� � ��� }d| }d| }d| }z�j�d�}W n ty/ t�d�j�j� Y dS w z
t ��
|�j�}W n t yM t� d�j�j� Y dS tyW d}Y nw �j�
� }d� �fd d
� } �z ||vr�||vr��jr��j}d| }d| }d| }||v �r��
�j�|��j� ||vr��j�|� � zt ��
�j�|��j�}
W n ty� tj d |�j�jdd � Y W dS w ||
k�r|j} t| t ��r| |v�r| |k�r| �d�d
} | s�t� d| �j�j� W dS d| }
|
|vr�|}
�j�| �j�|
�j� | |dd��sW dS ��� W S ||v �r��j�|�}t|tjtjf��s3t� d|t|�� nOzt |jj�}W n+ t �yf zt |j�}W n t �yc tj d|�j�jdd � Y Y W dS w Y nw t�d||� ||k�r}| |dd��s}W dS ��� W S W n) t�y� � t�y� } ztj d|�j�j|dd � W Y d}~dS d}~ww |�r�t�d|�j�j� �� � �j dd�S t� d|�j�j� dS )z�
Checkout the configured branch/tag
fetch_on_fail
If checkout fails perform a fetch then try to checkout again.
z refs/heads/r� r� r� z"HEAD not present in %s remote '%s'Nz,Unable to get SHA of HEAD for %s remote '%s'Tc
s� z4�j dd��# �j�| � |r!�j�� tj� W d � W dS W d � W dS 1 s-w Y W dS tyg } z'|jtjkrQt|jd�j � d�j
� d���t � d|j�j �j
� W Y d}~d S d}~ww )
zP
DRY function for checking out either a branch or a tag
rP r NTr� r r r� F)
r- r� rP �resetr ZGIT_RESET_HARDr r r/ rI r� rD rE )r� r� r� �� pygit2_idr� r r# �_perform_checkout s6
��
�� ����z*Pygit2.checkout.<locals>._perform_checkoutz5pygit2 was unable to get SHA for %s in %s remote '%s'rl r` zMpygit2 was unable to resolve branch name from HEAD ref '%s' in %s remote '%s')r� zG%s does not correspond to pygit2 Commit or Tag object. It is of type %szTUnable to resolve %s from %s remote '%s' to either an annotated or non-annotated tagzSHA of tag %s: %sFz-Failed to checkout %s from %s remote '%s': %sr� r� r� rp ) r� r� r� �lookup_referencer� rD r� rI r� r/ r� r� rE �listall_referencesrZ Zcreate_referencerX r@ r� r� �revparse_singler ZCommitZTag�typer� r rC r! rP )r� rO r[ Z local_refZ
remote_refZtag_refZ
local_headr� r� r� r� Zhead_refZ branch_nameZ remote_headZtag_objZtag_shar� r r� r# rP � s �
� �
� ��
�
��
��
���
����
� �zPygit2.checkoutNc sL zt jrg W S W n ty Y nw | jdur!t�dt| j� g S t� � � S )zX
Clean stale local refs so they don't appear as fileserver environments
Nz�The installed version of pygit2 (%s) does not support detecting stale refs for authenticated remotes, saltenvs will not reflect branches/tags removed from remote '%s')
r �GIT_FETCH_PRUNEr� rk rD r� �PYGIT2_VERSIONr� r~ r )r� �
local_refsr� r r# r � s � �
�
zPygit2.clean_stale_refsc C s� t j�d�tjjtj< d}t �| j�s t� | j�| _
d}n z t� | j�| _
W n t y< t
�t| j| j| j� | Y S w tjj�| j
jd�| _| �� t j�| jd�}t j�|�rgttd�krg| j
j� |� |S )z�
Initialize/attach to a remote using pygit2. Return a boolean which
will let the calling function know whether or not a new repo was
initialized by this function.
r FTra r z0.28.0) r� r� �
expanduserr ZsettingsZ search_pathZGIT_CONFIG_LEVEL_GLOBALr� r� Zinit_repositoryr� Z
Repositoryr� rD rE r� rz rI rs rt r| Zworkdirr� r+ �existsr� r r Zadd_file)r� r� r) r r r# r� � s$
� �zPygit2.init_remotec s� � �fdd�� t � }����}|s|S ����rCz|���� j}�j| }W n ty2 | Y S w t|tj�s;|S ��fdd�}ndd� }g }|rT� ||����� ��fdd�}|D ] }|� |||��� q]��
��rv|� ��
��� |S )zS
Get a list of directories for the target environment using pygit2
c sx t | �D ]5}|j�jvr
q�j|j }t|tj�sq|�tjj j
||j dd�� |r9� ||tjj j
||j dd�� qdS )z�
Traverse through a pygit2 Tree object recursively, accumulating all
the empty directories within it in the "blobs" list
Tr� N) r r� r� r@ r r� r� rs rt r� r| r
)r� �blobs�prefix�entryrl �� _traverser� r r# r� s" ����z"Pygit2.dir_list.<locals>._traversec r� rM r� r� rQ r r# r� ( r� z Pygit2.dir_list.<locals>.relpathc S r� rM r r� r r r# r� - r� c r� r� r� r� rQ r r# r� 4 r� z'Pygit2.dir_list.<locals>.add_mountpoint) r� r^ r r� r� r� r@ r r� r� r ) r� r� r2 r� r� r� r� r� rl r �r� r� r� r# rR s2
�
zPygit2.dir_listc C s | j �� }| �|�S )r� )r� r� r� r� r r r# rT ? s
z Pygit2.envsc
C s� | j jd }| j �� }i }| jdur| j|d< n | jdur!| j|_ztj|d< W n ty1 Y nw z
|jdi |��}W nL t y� } z@t
|�� � }d|v rat | jtj
�ratjd| j| jdd� n d |v rqtjd
| j| jdd� n tjd | j| j|dd� W Y d}~d S d}~ww z|d
}W n ttfy� |j}Y nw |dkr�t�d| j|| j� n t�d| j| j� | j �� }| j|d�} |s�||ks�| r�dS dS )r� r NZ callbacksZprunezunsupported url protocolz�Unable to fetch SSH-based %s remote '%s'. You may need to add ssh:// to the repo string or libgit2 must be compiled with libssh2 to support SSH authentication.Trl z+authentication required but no callback setzH%s remote '%s' requires authentication, but no authentication configuredz*Error occurred fetching %s remote '%s': %sF�received_objectsz&%s received %s objects for remote '%s'z %s remote '%s' is up-to-date)r� r )r� r� r� �remotecallbacksrk r r� r� r! �GitErrorr re r@ �KeypairrD rE rI r� r. r� r� r )
r� r� Zrefs_preZ fetch_kwargsr� r� Zexc_strr� Z refs_postr� r r r# r. G st
�
�� ����
��
z
Pygit2._fetchc s � �fdd�� t � }i }����}|s||fS ����rKz|���� j}�j| }W n
ty8 ||f Y S w t|tj�sC||fS ��fdd�}ndd� }i }|r\� ||����� ��fdd�}|� dg �D ] } |�
||| ��� qi|� d i �� � D ] \} }
|
|||| ��< q}||fS )
zG
Get file list for the target environment using pygit2
c s� t | �D ]Z}|j�jvr
q�j|j }t|tj�rItjjj ||j
dd�}|� dg �� |� t
�| |j
j�rH�j| |j
j j}||� di �|< qt|tj�r^� ||tjjj ||j
dd�� qdS )z�
Traverse through a pygit2 Tree object recursively, accumulating all
the file paths and symlink info in the "blobs" dict
Tr� r& r� N)r r� r� r@ r r� rs rt r� r| r
�
setdefaultr� r� r� �filemoderu r� )r� r� r� r� r� � repo_pathr� r� r r# r� � s* �� ���z#Pygit2.file_list.<locals>._traversec r� rM r� r� rQ r r# r� � r� z!Pygit2.file_list.<locals>.relpathc S r� rM r r� r r r# r� � r� c r� r� r� r� rQ r r# r� � r� z(Pygit2.file_list.<locals>.add_mountpointr& r� ) r� r^ r r� r� r� r@ r r� rq r� ry ) r� r� r& r� r� r� r� r� r� r� r� r r� r# rU � s4
� zPygit2.file_listc C s� | � |�}|s dS d}d}d} |d7 }|tkrd}nAz4|| }|j}t�|�r>| j|j j}tj j
j t j
�
|�|dd�}n| j|j }t|tj�rLd}W nW n tyZ d}Y nw qt|tj�rj|t|j�|fS dS r� )r^ r� r� r� r� r� r� ru rs rt r� r| r� r� r@ r r� r� r� r/ ) r� r� r� r� rl r� r� r� r� r r r# rV � s@
� � �� zPygit2.find_filec C �2 z| � | j�d|� ���jW S ty Y dS w )zk
Return a pygit2.Tree object matching a head ref fetched into
refs/remotes/origin/
r� N�r� r� r� r� r� r� r r r# r� s �� �zPygit2.get_tree_from_branchc C r� )zX
Return a pygit2.Tree object matching a tag ref fetched into refs/tags/
r� Nr� r� r r r# r� s
�zPygit2.get_tree_from_tagc C s. z| j �|�jW S ttttfy Y dS w )z<
Return a pygit2.Tree object matching a SHA
N)r� r� r� r� r. r? r� r� r r r# r� r� zPygit2.get_tree_from_shac C s` t td�kr tj| jd�| _| jsdd� }|| j_dS dS d| _| js.t� dt � d�� dS dS )z8
Assign attributes for pygit2 callbacks
z0.23.2rj c _ ri )NTr )�args�kwargsr r r# �_certificate_check) r� z2Pygit2.setup_callbacks.<locals>._certificate_checkNzdpygit2 does not support disabling the SSL certificate check in versions prior to 0.23.2 (installed: z2). Fetches for self-signed certificates will fail.)
r� r r ZRemoteCallbacksrk r� r% Zcertificate_check�warnings�warn)r� r� r r r# r� s ����zPygit2.setup_callbacksc s: d� _ tj�� j�r dS t� fdd�tD ��sdS � fdd�}� fdd�}� j�d �\}}}|s5d
}� j}|�� }|d v r?dS d
|v r�d }|� d
�d }||kr^t
� d� j � j
� t� j � |� _t� fdd�|D ��r�� fdd�dD �}dD ](\} }
|| } | dur�z
tj�| �s�||
| � W qw ty� ||
| � Y qww qwtj|� � _ dS � fdd�|D �} || � dS d|v �r d}t� fdd�|D ��}
t� fdd�|D �� }|r�dS |
r�|dkr� js�t
� d� j � j
� j � t� j � t�� j� j�� _ dS � fdd�|D �} || � dS t
� d � j � j
|� t� j � dS ) a�
Check the username and password/keypair info for validity. If valid,
set a 'credentials' attribute consisting of the appropriate Pygit2
credentials object. Return False if a required auth param is not
present. Return True if the required auth parameters are present (or
auth is not configured), otherwise failhard if there is a problem with
authenticaion.
NTc 3 s � | ] }t � |d �V qd S rM �r� r7 r� r r# rZ D s � z%Pygit2.verify_auth.<locals>.<genexpr>c s( t �d� j� jd�| �� t� j� dS )zU
Helper function to log errors about missing auth parameters
zPIncomplete authentication information for %s remote '%s'. Missing parameters: %srU N)rD r{ rI r� r| rJ )�missingr� r r# �_incomplete_authH s �z,Pygit2.verify_auth.<locals>._incomplete_authc s$ t �d| |� j� j� t� j� dS )zN
Helper function to log errors about missing key file
a SSH %s (%s) for %s remote '%s' could not be found, path may be incorrect. Note that it may be necessary to clear git_pillar locks to proceed once this is resolved and the master has been started back up. A warning will be logged if this is the case, with instructions.N)rD r{ rI r� rJ )�key_typer� r� r r# �_key_does_not_existU s � z/Pygit2.verify_auth.<locals>._key_does_not_existz://Zssh)r �file)r r �@r zJKeypair specified for %s remote '%s', but remote URL is missing a usernamec 3 � � | ] }t t� |d ��V qd S rM �r> r� r7 r� r r# rZ � � � c s g | ]}t � |d ��qS rM r� r7 r� r r# r: � s
��z&Pygit2.verify_auth.<locals>.<listcomp>)r r r r ))r_ r )r� r c � g | ] }t t� |d ��s|�qS rM r� r7 r� r r# r: � �
��r" )r r c 3 r� rM r� r7 r� r r# rZ � r� c 3 r� rM r� r7 r� r r# rZ � s �
�a Invalid configuration for %s remote '%s'. Authentication is disabled by default on http remotes. Either set %s_insecure_auth to True in the master configuration file, set a per-remote config option named 'insecure_auth' to True, or use https or ssh-based authentication.c r� rM r� r7 r� r r# r: � r� zEInvalid configuration for %s remote '%s'. Unsupported transport '%s'.)rk r� r� �isabsrz �anyr� r� re rB rD r{ rI r� rJ r �allr` r. r r� r ZUserPassr )r� r� r� Z transportr, �addressZrequired_paramsr Zkeypair_paramsrf r� Zkey_pathZ missing_authZ password_okZno_password_authr r� r# r� 6 s�
�
�
� ��
�
��
� �zPygit2.verify_authc C sB t jj�|d��}|�|j� W d � dS 1 sw Y dS r� )rs rt r& r' r( ru r� r r r# rn � s "�zPygit2.write_filero rp rM )rY rq rr rs r� r� rP r r� rR rT r. rU rV r� r� r� r� r� rn r� r r r� r# r� � s* �
[ ?LF) r� )r r| c @ s� e Zd ZdZddeedddfdd�Zedd� �Zed d
� �Z ed d � �Z
deefd
d�Z dd� Z dd� Z
dd� Zdd� Zd-dd�Zd.dd�Zd.d d �Zd.d d �Zd d!� Zd"d#� Zd/d%d&�Zd/d'd(�Zd)d*� Zd0d+d,�ZdS )1�GitBasez)
Base class for gitfs/git_pillar
Nr Tc C s� || _ |dur |nt| _| �� |dur| | _| _ntjj� | j d | j
�| _tjj� | jd�| _tjj� | jd�| _ tjj� | jd�| _ tjj� | j d d| j
�| _
tjj�| j� |rn| �|durf|ng |||� dS dS )a
IMPORTANT: If specifying a cache_root, understand that this is also
where the remotes will be cloned. A non-default cache_root is only
really designed right now for winrepo, as its repos need to be checked
out into the winrepo locations and not within the cachedir.
As of the 2018.3 release cycle, the classes used to interface with
Pygit2 and GitPython can be overridden by passing the git_providers
argument when spawning a class instance. This allows for one to write
classes which inherit from salt.utils.gitfs.Pygit2 or
salt.utils.gitfs.GitPython, and then direct one of the GitBase
subclasses (GitFS, GitPillar, WinRepo) to use the custom class. For
example:
.. code-block:: Python
import salt.utils.gitfs
from salt.fileserver.gitfs import PER_REMOTE_OVERRIDES, PER_REMOTE_ONLY
class CustomPygit2(salt.utils.gitfs.Pygit2):
def fetch_remotes(self):
...
Alternate fetch behavior here
...
git_providers = {
'pygit2': CustomPygit2,
'gitpython': salt.utils.gitfs.GitPython,
}
gitfs = salt.utils.gitfs.GitFS(
__opts__,
__opts__['gitfs_remotes'],
per_remote_overrides=PER_REMOTE_OVERRIDES,
per_remote_only=PER_REMOTE_ONLY,
git_providers=git_providers)
gitfs.fetch_remotes()
Nrn r� zenvs.p�hashZ
file_lists)ro �
GIT_PROVIDERS�
git_providers�verify_providerr� Z remote_rootrs rt r� r| rI � env_cache�
hash_cachedir�file_list_cachedir�cacheZverify_cache_version� init_remotes) r� ro r� �per_remote_overridesrS � global_onlyr r� r r r r# r� � s, 2����zGitBase.__init__c C sF |� dd�}t|�dkrd|fS |\}}tjj�|�s ||fS d|fS )aZ
Given a string determine if it is a url or a name and url combination.
Examples:
None, "https://github.com/saltstack/salt.git" == split_name("https://github.com/saltstack/salt.git")
"__env__", "https://github.com/saltstack/salt.git" == split_name("__env__ https://github.com/saltstack/salt.git")
r r_ N)rB r� rs rt �verifyrz )r� r� re Z maybenameZmaybeurlr r r# �
split_name
s zGitBase.split_namec C sR d}t �||� }r'|�� \}}}|�d�sd|� �}d|� d|� |� �}|S |S )a
Convert a remote to a URL
Remotes should be in url format with the exception of some ssh remotes
which can be in a `git@...` format. This method handles the special ssh
remote case by converting to `ssh://` style URLs.
z^([^@:/]+)@([^:]+):(.+)$r^ zssh://r� )�re�match�groupsr� )r� r� �patternr
r �hostr� rz r r r# �
remote_to_url.
s
zGitBase.remote_to_urlc C s. | � |�\}}| �|�}tjj�|�rdS dS )z7
Validate a remote repository config.
TF)r r rs rt r
rz )r� r� r, rz r r r# �validate_remote@
s
zGitBase.validate_remotec
sV t �|�}�fdd�tD �}�jtv r|t7 }n2|rJd�d�t��}�j� d�j� d�j� d�j� d|� d �
}�jd
kr@|d 7 }t� |� t
�j� i } t |�}
|
� t |�� |
D ]$} �j� d | � �} | �j
vrst� d
| � t
�j� t| �j
| �| | < qYg �_t|�}t|�D ]�}
t|
t�r�t|
�D ]} ��| �s�t�d| � |
�| � q�q�|
s�|�|
� n��|
�s�t�d|
� |�|
� q��j�j �j
|
| ||�j�j�}t|d��r^�j
d dkr�|jr�|�� i |_ |j � � D ]8\}}d|v �r&|d }|j � |g �� |� |dk�r&|j | �d� ||j!k�r&t�d�j|j"�j� ||_!q�g }|j �#� D ]}|�$|� �q.|j%� � D ]\} }| |v�rVd|v �rV|j � |d g �� | � �q<�j� |� q�i � �jD ]}� � |�&� g �� |j"� �qd� fdd�� D �}|�r�|D ]}t� d�jd�� | �� t
�j� �q�t'dd� �jD ���r���(� dS dS )z$
Initialize remotes
c s2 g | ]}� j � j� d |� � r� j� d |� ��qS )r, )ro rI r7 r� r r# r: [
s
��z(GitBase.init_remotes.<locals>.<listcomp>z{}rU z) authentication was configured, but the 'z' zb_provider does not support authentication. The providers for which authentication is supported in z are: �.rL r[ r, zNKey '%s' not present in global configuration. This is a bug, please report it.zFound bad url data %rr� �__role�minionr rX z�The 'base' environment has been defined in the 'saltenv' param for %s remote %s and will override the branch/tag specified by %s_base (or a per-remote 'base' parameter).c s g | ] }t � | �d kr|�qS )r_ )r� r7 )� cachedir_mapr r# r: �
� zqThe following %s remotes have conflicting cachedirs: %s. Resolve this using a per-remote parameter called 'name'.c s s � | ]}|j V qd S rM )r� r7 r r r# rZ �
s � z'GitBase.init_remotes.<locals>.<genexpr>N))rw rx r� r� r� r� r| rI rD r{ rJ r� r� ro rH r� rA r@ r} r r� �popr r r� r� r� r! r� rc ry r� r� rX r� �values�extendrv r� r� �write_remote_map)r� r� r rS r r� Zglobal_auth_paramsZmsg_auth_providersr� r� Z
global_valuesr� r0 r� Zrepo_objrc r� r Zall_envsZ env_namesrd r� Z
collisionsr� r )r r� r# r L
s�
�
����
�
�
�
�
�
� �� �
� �zGitBase.init_remotesc
C s\ zt �|� W n ty# } zt�d| j||� W Y d }~dS d }~ww t�d| j|� dS )Nz.Unable to remove old %s remote cachedir %s: %sFz%s removed old cachedir %sT)r r r rD rE rI r� )r� r� r� r r r# �_remove_cache_dir�
s ���zGitBase._remove_cache_dirc c sB � t �| j�D ]}|dv rqt j�tjj�| j|��r |V qd S )N)r� r� rk rj )r� r� r� r� r� rs rt r| )r� r1 r r r# �_iter_remote_hashes�
s ���zGitBase._iter_remote_hashesc C sV d}dd� | j D �}| �� D ]}||vr"| �tjj�| j|�p |�}q|s)| �� |S )zK
Remove cache directories for remotes no longer configured
Fc S s h | ]}|� � �qS r )r� )r8 r4 r r r# � <setcomp>�
r; z,GitBase.clear_old_remotes.<locals>.<setcomp>) r� r r rs rt r� r| r� r )r� ZchangeZ
remote_setr1 r r r# �clear_old_remotes�
s ��zGitBase.clear_old_remotesc C sp g }| j | jfD ]-}tj�|�r5zt�|� W q ty4 } z|�d|� d|� �� W Y d}~qd}~ww q|S )z(
Completely clear cache
zUnable to delete r6 N) r� r r� r� r� r r r r� )r� �errorsZrdirr� r r r# � clear_cache s "���zGitBase.clear_cacher� c C s� g }g }| j D ]6}|r+z t�|j|�sW qW n ty* t�|jt|��s(Y qY nw |j|d�\}}|�|� |�|� q||fS )z1
Clear update.lk for all remotes
r )r� �fnmatchrz r. r/ r
r )r� r� r� Zclearedr r� r r r r r# r
s"
� ��
zGitBase.clear_lockc
s| |du rg }nt |t�r|�d�}n
t |t�s t�d|� g }d}| jD �]}t|dd�}|r;|j|f|v s;||v �r;z�t j
�|� � �d � � fdd�t � � �D �}|D ][}||�
� kr]qTtjj
�� |�}tjj
�|d �}t j
�|�r�ztjj�|d
�� W d � n1 s�w Y W qT ty� } ztjd || d d
� W Y d} ~ qTd} ~ ww t�d|� qTt �� �D ]]}||�
� kr�q�tjj
�� |�}tjj
�|d �}t j
�|��r ztjj�|d
�� W d � n1 s�w Y W q� t�y } ztjd || d d
� W Y d} ~ q�d} ~ ww t�d|� q�|�� �rd }W q$ t�y: } ztjd| j|j| d d
� W Y d} ~ q$d} ~ ww q$|S )z�
Fetch all remotes and return a boolean to let the calling function know
whether or not any remotes were updated in the process of fetching
Nr< zLInvalid 'remotes' argument (%s) for fetch_remotes. Must be a list of stringsFr
r c s$ g | ]\}}}|st j�|� ��qS r )r� r� r� )r8 r� �subdirsr& �Zrepo_work_hashr r# r: ? s � �z)GitBase.fetch_remotes.<locals>.<listcomp>r_ r# z#Failed to make fetch request: %s %sTrl z Failed to make fetch request: %sz2Exception caught while fetching %s remote '%s': %s)r@ r/ rB rA rD rE r� r� r� r� r� r� �walkr� rs rt r| r� r& r' r r� r! rC rI )
r� r� �changedr� r
Zbranchesr� Zbranch_salt_dirZ
fetch_pathr� r r$ r# �
fetch_remotes' s�
�
� � ����� � �����
�����DzGitBase.fetch_remotesc C s� g }g }| j D ]4}|r+z t�|j|�sW qW n ty* t�|jt|��s(Y qY nw |�� \}}|�|� |�|� q||fS )z$
Place an update.lk
)r� r" rz r. r/ rD r )r� r� �lockedr r� r r r r r# rD � s"
� ��
z GitBase.lockc C sn ddd�}| � � |d< | j|d�rd|d< | jd dk}|d du s)tj�| j�s=tj�| j�}tj�|�s;t� |� d}|rn| j
dd �}t j j
�| jd
��}|�t j�|�� t�d | j� W d � n1 siw Y | j�d
d�r�t j jjd| jd | jdd��}|�|tddgdd�� W d � n1 s�w Y z t j�| j| j � W d S t y� Y d S w )aE
.. versionchanged:: 2018.3.0
The remotes argument was added. This being a list of remote URLs,
it will only update matching remotes. This actually matches on
repo.id
Execute a git fetch on all of the repos and perform maintenance on the
fileserver cache.
FrL )r& Zbackendr& �r� Tr r )� ignore_cacher� zWrote env cache data to %sNZfileserver_eventsZmasterZsock_dir)ro �listenr� �
fileserver�r� ) r r' ro r� r� r` r r� r� r� rT rs rt r& r' r( �payload�dumpsrD �tracerq �eventZ get_eventZ
fire_eventr r, Zreap_fileserver_cache_dirr rV r )r� r� ru Zrefresh_env_cacheZ env_cachedirZnew_envsr* r1 r r r# r� � sB
�� �
� �zGitBase.updatec C s dd� | j D �S )at
Returns a dictionary mapping remote IDs to their intervals, designed to
be used for variable update intervals in salt.master.FileserverUpdate.
A remote's ID is defined here as a tuple of the GitPython/Pygit2
object's "id" and "name" attributes, with None being assumed as the
"name" value if the attribute is not present.
c S s" i | ]
}|j t|d d�f|j�qS )r
N)r� r� r+ )r8 r� r r r# �
<dictcomp>� s ��z,GitBase.update_intervals.<locals>.<dictcomp>r) r� r r r# �update_intervals� s �zGitBase.update_intervalsc C s d| j � d�| jv r| jd| j � d� | _ne| j�| j � d��}|s6| jdd�r,d| _nO| jdd�r5d| _nEz|�� }W n tyK t|��� }Y nw || j vrdt
� d| j |d� | j �� t
| j � n|dkrp| �� rpd| _n |dkr{| �� r{d| _t| d �s�t
� d
| j � t
| j � d S d S ) z1
Determine which provider to use
� verified_� _providerT)�quietr r| z/Invalid %s_provider '%s'. Valid choices are: %srU r� z,No suitable %s provider module is installed.N)rI ro r� rq �
verify_pygit2�verify_gitpythonre r� r/ r rD r{ r| rJ r� )r� Zdesired_providerr r r# r � s< � �
�
�zGitBase.verify_providerFc s� � fdd�}t s|st�d� j� |� dS d� jvr dS g }t tk r2|�� j� dt� dt � d�� tjj �
d �sC|�d
� j� d �� |rV|D ]}t�|� qG|sT|� dS d� j d � j� d
�< t� d� j� dS )zX
Check if GitPython is available and at a compatible version (>= 0.3.0)
c �, t rd� jv rt�t� j� j� d S d S d S r� )r� r rD rE �_RECOMMEND_PYGIT2rI r r� r r# �
_recommend � �z,GitBase.verify_gitpython.<locals>._recommendzA%s is configured but could not be loaded, is GitPython installed?Fr| z: is configured, but the GitPython version is earlier than �
. Version �
detected.r zDThe git command line utility is required when using the 'gitpython' �
_provider.r4 r5 z gitpython %s_provider enabledT)
�GITPYTHON_VERSIONrD rE rI r �GITPYTHON_MINVERr� rs rt r� �whichro r� �r� r6 r; r rE r r� r# r8 � sF �
����� zGitBase.verify_gitpythonc s � fdd�}t s|st�d� j� |� dS d� jvr dS g }t tk r2|�� j� dt� dt � d�� ttk rE|�� j� d t� dt� d�� t t
d
d�s\t j j
�d �s\|�d � j� d
�� |ro|D ]}t�|� q`|sm|� dS d� jd� j� d�< t�d� j� dS )z�
Check if pygit2/libgit2 are available and at a compatible version.
Pygit2 must be at least 0.20.3 and libgit2 must be at least 0.20.0.
c r9 r{ )r@ r rD rE �_RECOMMEND_GITPYTHONrI r r� r r# r; - r< z)GitBase.verify_pygit2.<locals>._recommendzK%s is configured but could not be loaded, are pygit2 and libgit2 installed?Fr z7 is configured, but the pygit2 version is earlier than r= r> z8 is configured, but the libgit2 version is earlier than r� r zAThe git command line utility is required when using the 'pygit2' r? r4 r5 zpygit2 %s_provider enabledT)r� rD rE rI r �
PYGIT2_MINVERr� �LIBGIT2_VERSION�LIBGIT2_MINVERr� r rs rt r� rB ro r� rC r r� r# r7 ' sZ �
��������� zGitBase.verify_pygit2c
C s� t jj�| jd�}zGt jj�|d��5}t�� � d�}|�
d| j � d|� d�� | j D ]}|�
t jj
�|�� � d|j� d��� q*W d � n1 sJw Y W n
tyZ Y dS w t�d | j |� dS )
z*
Write the remote_map.txt
zremote_map.txt�w+z%d %b %Y %H:%M:%S.%fz# z_remote map as of r� z = Nz Wrote new %s remote map to %s)rs rt r� r| r� r& r' r �now�strftimer( rI r� r� r� r� r� r rD �info)r� Z
remote_mapr* � timestampr� r r r# r [ s$
��� �� �zGitBase.write_remote_mapc C s� t � � }t � � | dkrFz|j|d�W S tyE } z&|jtjkr,t �d� W Y d}~qtjd|j|j|j |dd� W Y d}~dS d}~ww t�d| j|j | j� dS ) z�
Common code for git_pillar/winrepo to handle locking and checking out
of a repo.
fetch_on_fail
If checkout fails perform a fetch then try to checkout again.
r� r� g�������?NzIError %d encountered while obtaining checkout lock for %s remote '%s': %sTrl z�Timed out waiting for checkout lock to be released for %s remote '%s'. If this error persists, run 'salt-run cache.clear_git_lock %s type=checkout' to clear it.)
rH rP r r r/ rI rD rE rI r� )r� r� rO rJ r� r r r# � do_checkouto s6
�
���zGitBase.do_checkout)Nr� rM �Frp )rY rq rr rs �PER_REMOTE_ONLY� GLOBAL_ONLYr� rv r r r r r r r r! r
r' rD r� r3 r r8 r7 r rM r r r r# r� � sF
�J
�
Z
2
%
+4r� c s� e Zd ZdZdZe�� Zddedddf� fdd� Z dedddfdd �Z
d
d � Z d d
d�Z d dd�Z
dd� Zdd� Zdd� Zdd� Zdd� Zd d � Z� ZS ) �GitFSz>
Functionality specific to the git fileserver backend
rL Nr Tc
s� |r
t jjjj�� nd}|r|| jvrKt�| �} t t
| �j ||dur$|ng |||dur-|nt ||d� |s<t
�d� | S t
�dt�� � | | j|< | S t
�dt�� � | j| S )a�
If we are not initializing remotes (such as in cases where we just want
to load the config so that we can run clear_cache), then just return a
new __init__'ed object. Otherwise, check the instance map and re-use an
instance if one exists for the current process. Weak references are
used to ensure that we garbage collect instances for threads which have
exited.
N)r rS r r� r z/Created gitfs object with uninitialized remotesz#Created gitfs object for process %sz$Re-using gitfs object for process %s)rs �extZtornadoZioloopZIOLoop�current� instance_map�object�__new__r~ rQ r� r rD r� r� r� )
r� ro r� r rS r r� r Zio_loopr� r� r r# rV � s(
�
�
z
GitFS.__new__c C s d S rM r )r� ro r� r rS r r� r r r r# r� � s zGitFS.__init__c C � | � |d�S )z@
Return a list of all directories on the master
rg �� _file_lists�r� �loadr r r# rR � s zGitFS.dir_listFc sv |st j�| j| j�}|dur|S t� }| jD ] � � �� }� j� � D ]}|�
|� q"|�
� fdd�|D �� qt |�S )zH
Return a list of refs that can be used as environments
Nc s g | ] }� � |�r|�qS r )rS r7 �r� r r# r: � s z GitFS.envs.<locals>.<listcomp>) rs r, Zcheck_env_cachero r r� r� rT r� r r� r% )r� r* � cache_matchr2 Z repo_envs�env_listr r\ r# rT � s
z
GitFS.envsrX c K s� ddd�}t j�|�r
|S tjjj| jd||dd�}tjjj| j||� d�dd�}tjjj| j||� d�dd�}tjjj| j||� d�dd�}t j� |�} t j� |�}
t j�
| �stzt � | � W n t ys t �
| � t � | � Y nw t j�
|
�s�zt � |
� W n t y� t �
|
� t � |
� Y nw | jD �]0} | �|�r�|�| �|�t j �s�q�tjj�|�s�|| �� vr�| js�q�|t| �|��d � �t j�} | �|�r�tjj�| �|�| �} | �| |�\}
}}|
d u r�q�d
d � }tj� ||� z@tjj � |d ��-}tjj� |� � �}||k�r$||d
< ||d< |||�W d � W S W d � n 1 �s/w Y W n t �yP } z|j!t!j"k�rF� W Y d }~nd }~ww tjj � |d�� W d � n 1 �sew Y t#�#|�D ]}zt �
|� W �qo t$�y� Y �qow | �%|
|� tjj � |d��
}|�&|� W d � n 1 �s�w Y zt �
|� W n
t �y� Y nw ||d
< ||d< |||� S |S )z�
Find the first file to match the path and ref, read the file out of git
and send the path to the newly cached file
r\ )r� �relr� T)Zsubdirz.hash.*z.hash.blob_sha1r� Nc S s |dur |g| d< | S )a!
Add a the mode to the return dict. In other fileserver backends
we stat the file to get its mode, and add the stat result
(passed through list() for better serialization) to the 'stat'
key in the return dict. However, since we aren't using the
stat result for anything but the mode at this time, we can
avoid unnecessary work by just manually creating the list and
not running an os.stat() on all files in the repo.
Nr� r )�fndr� r r r# �_add_file_stat,
s
z'GitFS.find_file.<locals>._add_file_statr4 r_ r� r# rH )'r� r� r� rs rt r
Z
clean_joinr� r r� r� r� r r r� r r� r� r� r� rT rZ r� �lstripr r| rV r, Z wait_lockr& r' r= r$ r r �globrC rn r( )r� r� r� r� r` rm Z hashes_globZ blobshadestZlk_fnZdestdirZhashdirr� r� rl Z blob_hexshaZ blob_modera r* Zshar� �filenamer r r# rV � s�
����
�
� � � �
� �� ��� �� ��zGitFS.find_filec s" d� v r � � d� ddd�}h d�}t� fdd�|D ��s+t�dd�|�� ��� |S |d s1|S |d
|d < � �d d
�}tj� |d �}t
j j �
|d��:}|�� d � |�| jd �}|rkt
j j �|�sk|�t�}|r{|r{t
j j�||�}||d < ||d< W d
� |S 1 s�w Y |S )zG
Return a chunk from a file based on the data received
r� r\ )ru rm > r� rc �locc 3 � � | ]}|� v V qd S rM r r7 �r[ r r# rZ h
� � z#GitFS.serve_file.<locals>.<genexpr>z<Not all of the required keys present in payload. Missing: %srU r� r_ rm �gzipN�rbre Zfile_buffer_sizeru )r r� rD r� r| �
differencerq r� r� �normpathrs rt r& r' r� r$ ro Z is_binaryr! r� Z gzip_util�compress) r� r[ r` r2 Zrequired_load_keysri Zfpathr* ru r rg r# �
serve_file^
s8
�
� � zGitFS.serve_filec
s� d� v r � � d� t� fdd�dD ��sdS d| jd i}|d }|d }| jd }tjj�| j� d |� d
|� ��}z#tjj� |d ��}|�
� |d < W d
� |W S 1 sWw Y |W S t yw } z
| j t j
krm� W Y d
} ~ nd
} ~ ww z t�tj�|�� W n t y� } z
| j t jkr�� W Y d
} ~ nd
} ~ ww tjj�|| jd �|d < tjj� |d��}|�|d � W d
� |S 1 s�w Y |S )zT
Return a file hash, the hash type is set in the master config file
r� c 3 rf rM r r7 rg r r# rZ �
rh z"GitFS.file_hash.<locals>.<genexpr>)r� rc )r\ Nrg r_ r� rc z.hash.rj ZhsumNrH )r r� ro rs rt r� r| r r& r' r$ r r r r� r� r� r/ Z hashutilsZget_hashr( )
r� r[ r` r2 r� r� Z lc_hash_typeZhashdestr* r� r rg r# � file_hash~
sR
�
�� ��� ���
��zGitFS.file_hashc C s� d|v r |� d� tj�| j�s+zt�| j� W n ty* t�d| j� g Y S w |d � tjj
d�}t j j�
| j|� d��}t j j�
| jd|� d��}t j�| j|||�\}}}|dura|S |r�t�d � t� i t� d
�} | jD ]I}
t j j�|d �s�|d | �� v s�|
jr�t�� } |
�|d �\} }
| d �| � | d �|
� | d
�|
�|d �� t� d|
j t�� | � qst | d �| d < t | d
�| d
< |r�t j� | j| ||� t�d� | � |g �S |d kr�i S g S )zL
Return a dict containing the file lists for files and dirs
r� zUnable to make cachedir %src z_|-z.pr z.wNz&Start rebuilding gitfs file_list cache)r& r� rg r& r� rg z9gitfs file_name cache rebuild repo=%s duration=%s secondsz)Finished rebuilding gitfs file_list cache)!r r� r� r� r r� r rD rE r� r� rs rt r| r, Zcheck_file_list_cachero r0 r� r� r� r� rT rZ rH rU r� rR Zprofiler� r% Zwrite_file_list_cacherq )r� r[ ZformZ lc_path_adjZ
list_cacheZw_lockr] Z
refresh_cacheZ
save_cacher2 r� �startZ
repo_filesZ
repo_symlinksr r r# rY �
sh
��
�
�
��
��
�
zGitFS._file_listsc C rW )zb
Return a list of all files on the file server in a specified
environment
r& rX rZ r r r# rU �
s zGitFS.file_listc C s g S )zF
Return a list of all empty directories on the master
r rZ r r r# �file_list_emptydirs�
s zGitFS.file_list_emptydirsc sv d|v r |� d� tjj�|d �s |d | �� vr i S d|v r(|d �d�� nd� | �|d�}� fdd�|�� D �S ) zQ
Return a dict of all symlinks based on a given path in the repo
r� rc r� r^ r\ r� c s i | ] \}}|� � �r||�qS r )r� )r8 r0 rF r- r r# r2 r z&GitFS.symlink_list.<locals>.<dictcomp>) r rs rt r� r� rT r6 rY ry )r� r[ r� r r- r# � symlink_list�
s
� zGitFS.symlink_listrN r� )rY rq rr rs rI �weakref�WeakKeyDictionaryrT rO rV r� rR rT rV rn ro rY rU rq rr r� r r r� r# rQ � s4 �3
�
l &>rQ c @ s& e Zd ZdZdZd dd�Zdd� ZdS )
� GitPillarz;
Functionality specific to the git external pillar
rV Tc C s� t � | _g | _| jD ]`}| j||d�}|durj|jdkr0t|d�r0| j�d�p.| j�d�p.d}n |j r7|j }n|j|j
kr@d}n
|� � }||j
krKdn|}|j re| �
|�rd|| j|�� < | j�|�� � q
|| j|< q
dS )z�
Checkout the targeted branches/tags from the git_pillar remotes
fetch_on_fail
If checkout fails perform a fetch then try to checkout again.
r� NrW r� rW rc rX )r Z pillar_dirsZpillar_linked_dirsr� rM r� r� ro rq r� rX r� ra �link_mountpointr� r� )r� rO r� rn r� Ztgtr r r# rP s,
�
�
��zGitPillar.checkoutc
C s& t jj�|�� |j�}t jj�|�� |�� ��t j
�}d}d}�zU|j ddd���A t t j
|�� dd��}||jkrRt�d|�� � t�d|� t�d|j� d }n�td
d � |d d
� D ��sht�d|� d }nmt jj�|�srd }ncz t jj�|�}W n ty� t�d|� d }Y nIw ||kr�t�d|||� z t jj�� r�|�d�s�t j�|�r�t�|� nt �|� W n ty� } z
t� d||� W Y d }~nd }~ww d}d }|�r#d }z t�|�� � W n t y� Y nw zt j� |�} t � | � t�d| � W n" t �y" } zt� d| |� W Y d }~W d � W dS d }~ww |�razt �!||� t�d||� W W d � W d S t �y` } zt� d|||� W Y d }~W d � W dS d }~ww W d � W d S 1 �snw Y W d S t"�y� t� d| j#|j$| j#| j#|j%dd�� Y dS w )zv
Ensure that the mountpoint is present in the correct location and
points at the correct path
Fr �
)r� r )� followlinksz2Results of walking %s differ from expected resultszWalk results: %szExpected results: %sTc s s4 � | ]}t jj�|d � otj�|d �V qdS )r N)rs rt r� �islinkr� r� r7 r r r# rZ D s
� "�
�z,GitPillar.link_mountpoint.<locals>.<genexpr>Nr` z-Linkdir parents of %s are not all directoriesz Failed to read destination of %sz=Destination of %s (%s) does not match the expected value (%s)z\\z;Failed to remove existing git_pillar mountpoint link %s: %sz#Successfully made linkdir parent %sz-Failed to os.makedirs() linkdir parent %s: %sz%Successfully linked %s to cachedir %sz-Failed to create symlink to %s at path %s: %sz�Timed out setting mountpoint lock for %s remote '%s'. If this error persists, it may be because an earlier %s checkout was interrupted. The lock can be cleared by running 'salt-run cache.clear_git_lock %s type=mountpoint', or by manually removing %s.r )&rs rt r� r| r� ra r� r r� r� r� r- rA r% rh rD r� r� ry �readlinkrC r� r� r� r� r r r � exceptionr r� r� rE �symlinkr rI r� r� )
r� r� Z
lcachelinkZ
lcachedestZ wipe_linkdirZ create_linkZ walk_resultsZldestr� Zldirnamer r r# rv - s� �
�
�� ��
��
�
���� �
�
��I ��Z�
��Z� �p�p�
� �zGitPillar.link_mountpointNrp )rY rq rr rs rI rP rv r r r r# ru s
!ru c @ s" e Zd ZdZdZi Zddd�ZdS )�WinRepoz�
Functionality specific to the winrepo runner
fetch_on_fail
If checkout fails perform a fetch then try to checkout again.
r] Tc C s8 i | _ | jD ]}| j||d�}|dur|| j |j< qdS )zN
Checkout the targeted branches/tags from the winrepo remotes
r� N)� winrepo_dirsr� rM r� )r� rO r� rn r r r# rP � s
��zWinRepo.checkoutNrp )rY rq rr rs rI r~ rP r r r r# r} � s
r} c C s� t �� }t� �dd�}t�| d �} | j�rJ| �� �rLt| � d��}g }d}d}zy|D ]N}t
j j �
|d��<}ztt
j j�|�� ��� �}W n tyP d}Y nw z
t
j j�|�� ��� }W n tyi d}Y nw W d � n1 stw Y q+||kr�||kr�|s�d|� d �} t�| � |�|||f� n|�|||f� W n ty� t�d
|� Y nw |D ]�\}}}zt �|� W nx t�y7 }
zk|
jtjkr�d |� d |� d
|� d|
� d� } t�| � nJ|
jtj k�rzt � |� W n; t�y }
zd|� d|� d
|� d|
� d� } t�| � W Y d}
~
nd}
~
ww d|� d|� d
|� d|
� d� } t�| � W Y d}
~
q�d}
~
ww d|� d|� d
|� d�} t�| � q�dS dS dS )z5
Clean up finalize processes that used gitfs
rP rQ z /gitfs/workz**/*.lkr r4 Nzgitfs lock file for pid 'a ' does not contain a machine id, deleting lock file which may affect if using multi-master with shared gitfs cache, the lock may have been obtained by another master recommend updating Salt version on other masters to a version which insert machine identification in lock a file.z gitfs lock file: %s not foundz<SIGTERM clean up of resources attempted to remove lock file z, pid 'z', machine identifier 'z$' but it did not exist, exception : r z*SIGTERM clean up of resources, lock file 'z', pid 'z2'was a directory, removed directory, exception : 'r z;SIGTERM clean up of resources, unable to remove lock file 'z', exception : 'z2SIGTERM clean up of resources, removed lock file ') r� r� rp rq �pathlib�Pathr� �is_dirrA rc rs rt r& r' r? r� r= r> r� r? rD r� r� �FileNotFoundErrorr r r r r r r ) r� r� rr rU Z
file_del_listZfile_pidZ file_mach_id� file_namer@ r� r� r r r# r� � s� � ��� �� ��
�
�� � ����� ������ �����
�� ���� �*r� )rrs r� rw rw r r" rc r� r� �loggingrt r� r r r� r r� r� rH rs � collectionsr r Zsalt.ext.tornado.iolooprs Zsalt.fileserverZsalt.utils.cacheZsalt.utils.configparserZsalt.utils.dataZsalt.utils.filesZsalt.utils.gzip_utilZsalt.utils.hashutilsZsalt.utils.itertoolsZsalt.utils.pathZsalt.utils.platformZsalt.utils.processZsalt.utils.stringutilsZsalt.utils.urlZsalt.utils.userZsalt.utils.versionsZ salt.configr r r� Zsalt.exceptionsr r r Zsalt.utils.eventr r
rp r rf rO rP r� r� r� r� rD r: r� � getLoggerrY rD r� r� � ImportErrorrt r� Z is_darwinr� rB Zsalt.utils.mac_utilsr r r� � __version__r@ rC r� �catch_warnings� simplefilterr� r� r r� rF Z
pygit2.ffiZ
pygit2.remoter$ r� Z maybe_stringr r� r� r� r@ r{ rA rE rG rH rJ rK r� r� r r� rQ ru r} r� r r r r# �<module> s. ���
�
� �
�
� ��
��6 Z � O q (