Yohohohohohooho | Sanrei Aya
Sanrei Aya


Server : LiteSpeed
System : Linux barito.iixcp.rumahweb.net 5.14.0-611.49.1.el9_7.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 21 16:39:08 EDT 2026 x86_64
User : elvh3918 ( 1528)
PHP Version : 8.2.31
Disable Function : mail
Directory :  /opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/__pycache__/

Upload File :
current_dir [ Writeable ] document_root [ Writeable ]

 

Current File : //opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/__pycache__/db.cpython-311.pyc
�

$�&j�-��|�ddlZddlZddlZddlZddlmZmZddlmZmZm	Z	m
Z
mZmZm
Z
mZmZmZddlmZddlmZddlmZddlmZddlmZd	Zed
zZdZe��Zejd��Z Gd
�de��Z!d d�Z"d�Z#d�Z$d�Z%d�Z&d�Z'd�Z(d�Z)d�Z*d!d�Z+d�Z,d�Z-d�Z.ej/defd���Z0dS)"�N)�datetime�	timedelta)
�Column�Boolean�DateTime�Index�Integer�String�
create_engine�event�func�text)�declarative_base)�	Inspector)�Session)�close_all_sessions)�
DatabaseErrorz/var/lve/ssa.dbz.old��cleanup_old_datac���eZdZdZdZeddd��fZeed���Z	ee
d�	��Zee
d�	��Zeed�	��Z
eed�	��Zeed�	��Zeed�	��Zeed�	��Zeed�	��Zeed�	��Zeed�
��ej��d���Zeed�
��ej��ej�����Zd
S)�
RequestResultay
    Describes processed request stored in database file.

    E.g.
    {
        "timestamp": "1650008727",
        "url": "http://mydomain.com/index.php",
        "duration": 162077,
        "hitting_limits": false,
        "throttled_time": 0,
        "io_throttled_time": 0,
        "wordpress": true
    }

    Note: created_at, updated_at is saved in local TZ format
    �
scrape_result�ix_scrape_result_domain_path�domain�pathT)�primary_keyF)�nullable)�timezone)�server_default�index)�onupdaterN)�__name__�
__module__�__qualname__�__doc__�
__tablename__r�__table_args__rr	�idr
rr�	timestamp�durationr�is_slow_request�hitting_limits�throttled_time�io_throttled_time�	wordpressrr
�now�
created_at�
updated_at���=/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/db.pyrr(sk�������� $�M�	��,�h��?�?��N�
���T�	*�	*�	*�B�
�V�F�U�
+�
+�
+�F��6�&�5�)�)�)�D���w��/�/�/�I��v�g��.�.�.�H��f�W�u�5�5�5�O��V�G�e�4�4�4�N��V�G�e�4�4�4�N���w��7�7�7����w��/�/�/�I�����$�/�/�/����
�
�RV�W�W�W�J�����$�/�/�/�(�$�(�*�*�U]�UY�U]�U_�U_�`�`�`�J�J�Jr4r�'c��tj��tt���z
}d}	t	|��5}|�t
d��||d���}|j}||z
}|dkr
	ddd��n	ddd��n#1swxYwY�nt�	d|��t|��dS)a
    Removes outdated records from database, saving disk space.
    Deletions are batched to avoid loading the entire database into
    the cgroup's page cache, which causes OOM on large databases.
    Runs VACUUM after deletion to reclaim disk space.
    )�daysrTzrDELETE FROM scrape_result WHERE rowid IN (SELECT rowid FROM scrape_result WHERE created_at < :cutoff LIMIT :batch))�cutoff�batchNzCleanup deleted %d old records)r�todayr�RETENTION_TIME_DAYS�
session_scope�executer�rowcount�logger�info�vacuum_database)�engine�
batch_size�
n_days_ago�
total_deleted�session�result�deleteds       r5rr\s5����!�!�I�3F�$G�$G�$G�G�J��M�	�
�6�
"�
"�	�g��_�_�T�\�&�&�%�z�:�:�<�<�F��o�G��W�$�M��!�|�|��	�	�	�	�	�	�	��	�	�	�	�	�	�	�	�	�	�	����	�	�	�	�	��K�K�0�-�@�@�@��F�����s�:B�B�Bc�j�tj�tj�t����}tj|tj��r6|�d|�d���t�	d|��dSt�
d||��dS)u�
    Point SQLite's temp store at the database directory (on disk) for the
    given operation. SQLite spills large sorts / temp B-trees and VACUUM
    copies to the system temp directory, which can be tmpfs (RAM-backed).
    When it is, the spill is charged against the MemoryMax=1G cgroup and
    OOM-kills the agent on large databases — the same hazard for VACUUM
    and for index builds / ORDER BY sorts (CLPRO-2819, CLPRO-3077).
    �PRAGMA temp_store_directory = '�'z&Set SQLite temp_store_directory to: %szJDirectory %s not writable. %s will fall back to the default temp location.N)�osr�dirname�abspath�SSA_DB�access�W_OKr>r@rA�warning)�cursor�	operation�db_dirs   r5�_redirect_temp_store_to_diskrWss����W�_�_�R�W�_�_�V�4�4�
5�
5�F�	�y����!�!�1����B��B�B�B�C�C�C����<�f�E�E�E�E�E����
�$�i�	1�	1�	1�	1�	1r4c�H�|���}|j}	d|_|���}t|d��t�d��|�d��t�d��|���nA#t$r4}t�	dt|������Yd}~nd}~wwxYw||_|���dS#||_|���wxYw)z�
    Run VACUUM command to reclaim disk space after deletions.
    VACUUM must be run outside of a transaction, so we use raw connection.
    N�VACUUMzStarting VACUUM operation...zVACUUM completed successfully.zVACUUM failed: )�raw_connection�isolation_levelrTrWr@rAr>�close�	Exception�error�str)rC�raw_conn�original_isolation_levelrT�es     r5rBrB�s!���$�$�&�&�H�'�7���#'�� ����"�"��$�V�X�6�6�6����2�3�3�3����x� � � ����4�5�5�5����������1�1�1����/�s�1�v�v�/�/�0�0�0�0�0�0�0�0�����1����$<�� ����������$<�� ����������s0�BB&�%D�&
C$�0*C�D�C$�$D�D!c�4�|���}	|���}t|d��t�d��|�d��|���t�d��|���n?#t$r2}t�	dt|����Yd}~nd}~wwxYw|���dS#|���wxYw)u�
    Create the composite (domain, path) index on existing databases.

    The schema is only materialised by ``create_all()`` in
    ``create_db_if_not_exist()``, which runs solely when the DB file does
    not yet exist — there are no migrations. So a model-level index reaches
    new installs only, while the multi-GB databases that actually OOM never
    get it. This explicit ``CREATE INDEX IF NOT EXISTS`` closes that gap; it
    is idempotent and a cheap no-op once the index exists (CLPRO-3077).

    Building the index on a large table sorts it, so we first redirect the
    temp store to disk to avoid the very tmpfs spill we are trying to fix.
    zCREATE INDEXz/Ensuring ix_scrape_result_domain_path exists...zVCREATE INDEX IF NOT EXISTS ix_scrape_result_domain_path ON scrape_result(domain, path)z.Index ix_scrape_result_domain_path is present.zFailed to ensure indexes: %sN)rZrTrWr@rAr>�commitr\r]r^r_)rCr`rTrbs    r5�ensure_indexesre�s���$�$�&�&�H�
����"�"��$�V�^�<�<�<����E�F�F�F����
-�	.�	.�	.�	���������D�E�E�E����������=�=�=����3�S��V�V�<�<�<�<�<�<�<�<�����=����	������������������s0�BB,�+D�,
C(�6(C#�D�#C(�(D�Dc�f�t|��s!tj�|��dSdS�N)�
is_db_present�Base�metadata�
create_all)rCs r5�create_db_if_not_existrl�s8���� � �)��
� � ��(�(�(�(�(�)�)r4c���tj�t��sdSt	j|��}d�|���D��}t|��dkS)NFc��g|]}|��Sr3r3)�.0�tables  r5�
<listcomp>z!is_db_present.<locals>.<listcomp>�s��
G�
G�
G��e�
G�
G�
Gr4r)rMr�isfilerPr�from_engine�get_table_names�len)rC�database_inspection�tabless   r5rhrh�s^��
�7�>�>�&�!�!���u�#�/��7�7��
G�
G�!4�!D�!D�!F�!F�
G�
G�
G�F��v�;�;��?�r4c�0�|�d��dS)NzPRAGMA journal_mode = WAL)r>)�	dbapi_con�
con_records  r5�setup_wal_moder{�s��
���1�2�2�2�2�2r4c�p�tj�tj�t����}tj|tj��sdS	|�d|�d���dS#tj	$r'}t�d||��Yd}~dSd}~wwxYw)uZ
    Keep SQLite's temp spills (sort / GROUP BY temp B-trees) on disk rather
    than the default temp dir, which may be tmpfs (RAM-backed) and thus
    charged against the MemoryMax=1G cgroup. ``temp_store_directory`` is a
    process-wide setting, but applying it on every connect is cheap and
    guarantees it is in force before the daily routine runs (CLPRO-3077).

    Complements the (domain, path) index: the index removes the per-domain
    ORDER BY sort entirely, this catches whatever spill remains (e.g. the
    GROUP BY on the computed hour). Best-effort — never block a connection.
    NrKrLz,Could not set temp_store_directory to %s: %s)rMrrNrOrPrQrRr>�sqlite3�Errorr@rS)ryrzrVrbs    r5�setup_temp_store_dirr�s����W�_�_�R�W�_�_�V�4�4�
5�
5�F�
�9�V�R�W�%�%����"����E�F�E�E�E�F�F�F�F�F���=�"�"�"����E��q�	"�	"�	"�	"�	"�	"�	"�	"�	"�����"���s�$A?�?B5�B0�0B5c����dt���|r��d���fd�}td|d���}tj|dt��tj|dt
��t
|��|S)Nzfile:z?mode=roc�0��tj�d���S)NT)�uri)r}�connect)�connection_strings�r5�<lambda>z!_setup_database.<locals>.<lambda>�s���g�o�&7�T�B�B�B�r4�sqlite:////F)�creator�echor�)rPrr�listenr{rrl)�readonlyr�rCr�s   @r5�_setup_databaser��s����(��(�(���;�0�:�:�:��B�B�B�B�G�
��w�U����F�
�L���N�3�3�3�	�L���$8�9�9�9��6�"�"�"��Mr4Fc� �t|��Srg)r�)r�s r5�setup_databaser��s���8�$�$�$r4c���tj�t��r-t	��tjtt��t��}|���tdt����}	t|��5}t|��5}|�t���
��t||��ddd��n#1swxYwYddd��n#1swxYwYn#t$rYnwxYwttdztdzfD]5}tj�|��rtj|���6dS)z�
    Restore database by establish connections to old and new databases,
    merge data to new one if possible and delete old one.
    r�Nz-walz-shm)rMr�existsrPr�replace�
OLD_SSA_DBr��disposerr=�queryr�first�!merge_unharmed_data_from_databaser�remove)rC�
new_engine�
old_engine�session_old�session_newrs      r5�restore_databaser��s���

�w�~�~�f���'�����
�
�6�:�&�&�&��!�!�J�
�N�N�����9�Z�9�9�:�:�J�
�
�:�
&�
&�	H�+�}�Z�7P�7P�	H�T_����m�,�,�2�2�4�4�4�-�k�;�G�G�G�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H����	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H����	H�	H�	H�	H����
�
�
���
�����V�f�_�f�v�o�>����
�7�>�>�$���	��I�d�O�O�O���sZ�D�D�+=C4�(D�4C8	�8D�;C8	�<D�?D�D�D�D�D�
D$�#D$c��d}d}	|�t���|���|��}	|���}|sdS|D]}|�|���nv#t$rit|��D]V}	|�||z�����}|sn&|�|���G#t$rY�SwxYwYnwxYw|�	��||z
}��)z_
    Scrape all unharmed records from malformed database and merge them into new database.
    r�
TN)
r�r�offset�limit�all�merger�ranger�rd)	r�r�r�rDr��records_to_save�record�pos_in_batch�record_to_saves	         r5r�r�sN���F��J���!�!�-�0�0�7�7��?�?�E�E�j�Q�Q��	�#�i�i�k�k�O�"�
���)�
*�
*���!�!�&�)�)�)�)�
*���	�	�	� %�j� 1� 1�
�
���%*�\�\�&�<�2G�%H�%H�%N�%N�%P�%P�N�)�����%�%�n�5�5�5�5��$�����D�������	����	�������*���%sH�A:�A:�:C-�,C�C-�C�C-�
C'�$C-�&C'�'C-�,C-c���tj�t��r=tj�t��rtjt��	t
|��5}|�td����}|�	��}|dddkcddd��S#1swxYwYdS#t$rYdSwxYw)z�
    Try integrity check of database file to see if it is malformed.
    If database unable to execute it, will also count as malformed.
    zPRAGMA integrity_checkr�okNT)rMrr�r�rPr�r=r>r�fetchallr)rC�dbrH�errorss    r5�is_malformed_databaser�1s��

�w�~�~�j�!�!��b�g�n�n�V�&<�&<��
�	�*�����
�6�
"�
"�	(�b��Z�Z��%=� >� >�?�?�F��_�_�&�&�F��!�9�Q�<�4�'�	(�	(�	(�	(�	(�	(�	(�	(�	(�	(�	(�	(����	(�	(�	(�	(�	(�	(�������t�t����s7�#C�2AC�:C�C�C�C�C�
C"�!C"�returnc#��K�t|���}	|V�|���n#|����xYw	|���dS#|���wxYw)zF
    Provide a transactional scope around a series of operations.
    )�bindN)rrd�rollbackr\)rCrGs  r5r=r=As�����
�6�"�"�"�G���
�
�
����������������
����	�
	�
�
��������
�
�������s�-�A�A�A�A4)r6)F)1rM�
contextlibr}�loggingrr�
sqlalchemyrrrrr	r
rrr
r�sqlalchemy.ext.declarativer�sqlalchemy.engine.reflectionr�sqlalchemy.ormr�sqlalchemy.orm.sessionr�sqlalchemy.excrrPr�r<ri�	getLoggerr@rrrWrBrerlrhr{rr�r�r�r�r��contextmanagerr=r3r4r5�<module>r�s���
�	�	�	�������������(�(�(�(�(�(�(�(�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�8�7�7�7�7�7�2�2�2�2�2�2�"�"�"�"�"�"�5�5�5�5�5�5�(�(�(�(�(�(�	��
�f�_�
��������	��	�-�	.�	.��1a�1a�1a�1a�1a�D�1a�1a�1a�h����.1�1�1�&���6���@)�)�)�
���3�3�3�"�"�"�,���%�%�%�%����8���6
�
�
� ���W��������r4

Yohohohohohooho | Sanrei Aya